By all means use the hand loads. Not telling you not to. And please don't take my opinion that you are wrong for doing it.For me, I will carry factory, just based on the advice I have gotten from the LE field based on the localities that I have lived in. Solid arguements can be made both ways for sure. lets put them out there and folks can decide for themselves.
As far as Rifle ammo, I am with you. I have not had a single problem with my handloads. No misfire...etc. You are hard pressed to find factory rifle ammo in my house for the same reasons you listed for pistol.