Don't know that it's your problem, but try making a shot and let the barrel cool down before firing the next shot. Do this for a series of 5 shots and see if it helps. When everything else remains constant but your bullet impact shifts it may mean your barrel is heating to a point that affects your shot placement.

Just a thought. Good luck.