Sounds like you might have an inconsistent anchor.
Try a two point anchor like Rich Welch advises with the rear feather of the arrow touching the tip of your nose and the thumb of your draw hand touching your earlobe at anchor.
The grip, as mentioned by Bigcountry above, can cause problems too.