Sounds like those first 20 shots were fired with ZERO cleaning! Start from scratch. Clean that bore until it is COMPLETELY copper free. That will probably take at least an hour of constant cleaning with sweets, soaked patch followed by dry patches,keep repeating this process until it comes out completely clean with no blue.
Now you are ready to start over, cleaning between each shot or at least every 3rd shot for 40 or 50 rounds.
as far as groups going low, most guns will shift zeroes as the barrel heats up. Which way they shift has never been predictable for me. But, once they start shifting that particular gun will shift in this same direction every time. Ive seen them walk in all directions, but each gun will walk the same way each time. That much is predictable.