Were you swabbing the barrel between shots? This is very important when sighting in. Also you prepared the rifle properly to shoot in the first place I am sure by cleaning any oils or lubes out of the barrel before starting.

What were your swabbing procedures and did you change them during the day? I watched my friend swab his rifle one afternoon. The first time he ran one wet patch and then one dry patch. The next time he ran two wet patches and really scrubbed the bore, then ran two dry patches. Any little thing you change can make a difference. I am not saying this is what happened in your case, I am just looking for possible causes...

Have you made sure that the load you are shooting is the most accurate and consistent projectile and load for your rifle that you could find? Some days one projectile will shoot excellent. The next day out, you wonder why you ever bought the things to shoot because they are all over the place. Have you had good shooting results with the load you were shooting these days. Or are you still experimenting?

Were you letting the barrel cool between shots? Many times a barrel in warm temperatures can overheat. They can really get hot. Just wrap your hand round them and see if they are hot to the touch. If they are, let them cool. When they over heat they will often times start throwing rounds all over the paper.

Did you change anything? Same container of powder. Same charge. Same sabots. Same primers. Again, the more consistent you can be the more likely the shots will be closer to the first and last.

What kind of rifle are we talking and what powder and projectile? Maybe someone had similar results with the one that is giving you fits. Also until you are sure the projectile and charge you are shooting is the best for the rifle, do not be so quick to move the sights. I've owned my Trade Rifle for 7 months and have a good load working, but have yet to move the sights for final adjustment.