It sounds like you're either torquing or peeking. If you're getting good groups with "flyers" once in a while, then continue to work on your form. Have someone watch you to see what you are doing. You might even be flinching a little. Or, if there is no one to help you,you could set up a video camera and then watch it to see what you are doing.

Perhaps if you consciously "follow through" you might find that your "flyers" will decrease.