.... some barrels are just rough. When you fire bullets down the rough barrel, you are "filling in" the (hopefully) microscopic pits and tool marks, which smooths the bore.... and groups shrink. That is why some people will not clean their bore during hunting season, once they have it "seasoned" with the right number of previous shots. It varies from rifle to rifle... I've heard of some that required 8-10 shots before it started grouping... some will group after the first "fouling" shot. I have heard that lapping your barrel will improve this situation, but I have no first-hand knowledge of this. I probably wouldn't fire-lap it, though, from what I've read about that process.
If your rifle is repeatable on the distance out of your first shot, then you should just allow for it on the hunt... if you have cleaned your bore before hunting.

"Progress, far from consisting in change, depends on retentiveness..those who cannot remember the past are condemned to repeat it." Santayana