As soon as the charge goes off, it starts pushing the bullet, slug, ball, what have you, down the barrel and pushing the gun backwards.
How you hold it makes a difference. When my husband taught me to shoot, he had me hold the gun with different grip. If you hold it too tightly, your muscle tremors, which you can't actually feel, can throw it off. If you hold it too loosely, it can bounce around when the charge goes off. You have to hold it snugly but not too tight and straight back against your shoulder. Your support of the forearm has to form a straight line back to your shoulder so when the charge goes off, the gun goes straight back.
My husband learned to shoot as a kid back in the Ozarks. But they taught him the same stuff in the Army.