Ther are a couple of things that can cause this. One: Your draw length may be too long and you have trouble staying right at full draw. You may not realize it but most people draw back to the stops and then when setling in they creep forward, some as much as 1/2".

The second is form related. Do you get the urge to see your arrow go or see where it hits? If you do then it's possible that as you release you "collapse" pulling your head away from the string while at the same time pushing the bow to the right. All this in an effort to see the arrow go.

And then there's the possiblity it's a combination of the two. And I'd bet it is. None the less, it's not necessary to see your arrow in flight. The only thing you should be mentally conscious of is aiming the bow. To that end, if you execute the shot right then the arrow will be where your sight pin was when the bow went off. You have to have confidence in your equipment and your abilities with it.

Now, this isn't something you learn overnight. It takes a lot of mental work long before any hunting season comes along. It takes hours and hours during the spring and summer shooting SPOTS or no target at all (blind baling).You have to train your brain which in turn trains the body.Once you get your form down and your confidence up then I think you'll find that you won't be shooting right or left on any given day.