the peep not rotating consistently will give you fits and you won't even realize it till you miss the X. It'll turn in crooked, basically giviing you a distorted window, and your eye will naturally try to center that in the sight housing and throw you off. You may even actually move the string closer or farther from your face with your release hand(left and right, not back and forth)to compensate for the distorted peep window and not even be realizing it.

Check your basic stuff: fall away rest adjustment, cam timing etc.