Could the flyers be from inconsistent ignition? In other words the sabot is moving from the primer before the charge ignites, or there is a difference in the pressure with which the sabot seals the powder charge between shots?

BP and subs are relatively immune to this, however smokeless is highly dependent upon consistent pressure at ignition to maintain uniform combustion and accuracy. This is why precision riflemen fret so much about neck tension in cartridge guns.