I think your Dad might be right. There are a lot of people that do not shoot well off hand. Underclocked has some good ideas with the hidden load. I did the same thing when I was shooting revolver, and solved it the same with with empty chambers.

Another thing that might be happening when you shoot off hand is, you're taking to long to get target acquisition and because of this wandering the sight and trying to spot your trigger pull. When I shoot free hand I shoot much better with a straight pull, fast target acquisition, and fire. When I take my time, I start to swing and it makes for poor accuracy.