If you haven't seen this exact thing happen about 5 times, it is my opinion you should go back to the range with a cold clean rifle, and try it again. It very well may be you will notice 2 groups forming. Just saying it is possible. The first two shots may be a group, maybe not. It may turn out you can sight your rifle in for the first two shots from a clean barrel.

Yes i understand there are many that need to shoot more than two shots on a day of hunting, but it seems most of us don't.