I hate to say this, but I've seen this a few times and it's always been the shooter.
There are a couple of possibilities here. Rememeber, I don't know you, so I'm just throwing it out there. Not trying to slam you.
1. Most likely... You are positioning yourself different in relation to the scope at different magnifications. I myself have done this. Confused me till I figured it out. You will see one sight pic at 9x and a different one at 4x with your head in the exact same spot. The natural tendency is to adjust your cheek weld to mirror the image you think you should have. This is a mistake. You need to sight in at a lower magnification to eliminate some target movement, and then switch to a higher magnification if you want. But you gotta keep your eye relief and cheek weld the exact same to achieve the same results.
2. probable... You're flinching. You've already said it bothers you. Gotta fix that no matter what the problem turns out to be.
3. possibly... Your scope is not tightened down enough and is moving just enough to not visibly notice, but enough to throw the shots.
4. least likely... Your reticles are loose and adjusting the scope is slightly moving them. This is highly unlikely because the recoil would cause more movement than you touching it, but stranger things have happened.
If I were you, I'd reset the scope mount, reset the scope, and start all over. Sight in at lowest magnification. If target aquisition is a problem for you at the lowest mag, then sight in at shorter distances. Like 50 yards. Know the trajectory of your bullet, put a sticker where it should strike if you sight in dead center, and adjust to the sticker. You can then test at 100 yards. Should be dead on. Then you can test at higher mag, should still be dead on. Practice dry fire at home until you are dead certain that your eye relief and cheek weld are perfect. Then move on to the range. You just gotta get over the recoil thing. Dry fire can help you here, but a lot of shooting (correctly) will cure it. You could just be flinching. Flinching can account for quite a bit of drift. Some people flinch more the more they shoot due to shoulder fatigue. If that's a huge problem, get a shoulder pad to help and a better butt pad. Make shooting as comfortable as possible so you practice good habits.
Good luck. Post any results. I'm interested to here what it is. I could be totally wrong and if anyone sees any problems with what I've suggested, chime in. I don't want to make his situation any worse, that's for sure.