Since you eliminated the arrow, how about this: is it the same shot each time that drifts? For example, always the last?

another thought is that as you stack the arrows in, the spot you are aiming at changes its look, this might make you aim off a bit.