Is your rest centered? I would start with center as mentioned earlier, my bet is you are not center and you are having to adjust your sight to compensate.