Dirt, Now I am surmising this as I don't use pellets. But with pellets you can probably push too hard and crush the pellets under the projectile thus making some of it 'loose' powder. This would probably change the burning rate from load to load depending on the amount of load pressure and varying the depth the ramrod is down the bore. Which would in turn be detrimental to accuracy.
IMO, with the right bullet or bullet/sabot combo, a firm steady pressure should be all that is needed to seat it over the charge. You will feel it make solid contact with the charge. That's when you make your reference mark on the ramrod for consistancy (and to check while in the field).