Another thought. If you had to crank your scope way off center to get it zeroed--that is, you are close to running out of elevation adjustment--and then have to move left or right, that could cause the symptoms you describe. A zeroed scope should have both windage and elevation pretty close to the center of their adjustment ranges.

cogito, ergo armor.