Sounds to me like you are left-eye dominant while shooting right handed or vice-versa.
It happened to me when I first started to learn how to shoot.
try using a peep sight and squinting the eye opposite the type of bow you have (RH-LH)
rh bOW CLOSE LEFT eYE. YOU SHOULD FIND THAT YOUR SIGHT NEEDS TO GO BACK THE OPPOSITE WAY YOU HAVE BEEN MOVING IT!