From my limited experience it's common to have the POI stray as the barrel heats up. In my guns, I can usually get a good group with three consecutive shots, but they seem to stray quite a bit after 3 or 4. For hunting it's just fine.

It seems like your's is taking this to the extreme. I'd consult a good trusted gun smith and consider free floating, and/or bedding depending on what he thinks is going on.

Also, if that first shot is through an oiled barrel, it always makes a significant POI difference from the next shot, cold or warm, in my experience. I never oil my barrel except for storage. Then when I shoot it next, I run a dry bore snake through it and also shoot a "fouling shot" before I check grouping.