You don't want to try and hold the muzzle down. What you want is consistency, use proper form and try to do it exactly the same every time.
The rifle is most always more accurate than the shooter is. When things go wrong or your point of impact shifts, most times it is because the shooter has introduced a variable, bad form or made some other mistake.
Theoretically, you could hold your rifle upside down, dial in your scope and shoot good groups, if you did it the same every shot.
Ammo type, bullet weight, outside temperature and other factors also play a role. Either singly or a combination of factors.
You also have to make sure your equipment is OK, scope tight, bore clean or dirty (one or the other). Again consistency is the key.
Tensing up is one of the major mistakes of a beginning shooter. Relax be consistent and let the rifle do most of the work.
A lot of people talk now on days about five shoot groups. IMO three shot groups are the way to shoot, after three rounds barrel temps play a larger roll in point of impact.