TN ! I see each group has a flyer ? As to whether it is you or the loads?
Try this : Take a magic marker and number each case and keep track of where each one hits. Keep the flyer cases away from your next load and shoot session to see if it is the case causing it {The flyer}.
This would be tough if you don't have a spotting scope though to see where each hits on the target!