Definitely sounds like something you are doing rather than your bow to me. Maybe you are gripping different or anchoring different. you have to move your sight when you are shooting one way and then you go back to how you were shooting/holding before and then you have to move your sight back to where you started. No coincedence you keep ending up back at your original sight mark.