Do you mean to say that you actually loosened, or even removed your cable guard?

If so, more than likely it is not tight, and is moving ever so slightly. That would account for your groups to keep moving back to the right.