I don't know about you but I have the same problem. I attribute it to my age, 61. With a scope I also can nail the bullout to hundreds of yards. But open sights kill me. I seem todo alright with a peep sight on my 1903a3 though.
Don't know your age, but that is my problem and it sounds like what you are experiencing. If so, wellcome to the senior citizen gang!!!