Im, guessing that it will be your form and grip position. When you draw slowly you would have a more stable and consistent grip throughout the draw but when you yank it back it is likely to change during the draw. You shouild set your grip and then leave it like that always. Never change your grip during you draw or during your aiming period.

You want your lifeline of your palm to reside on the middle of the grip. Your fingers should not touch the bow, but simply curl around and hang. You want your hand to be at a 45 degree downward angle to the grip. To get it consistent each time and put it into your subconcious, you should do some blank bale shooting before and after you practice. But, make sure that you work on only your grip. Then once that is consistent you can move on to the next truoble area of your form.