Does it come return to the original POI after the barrel cools down to ambient? If so I would be suspect of the bedding/stock channel. I know you said it is free floated to the barrel nut but how is after the stacked shots? Check for rub marks on the stock or barrel that would indicate touching. Could also be improper torque on the action screws, check the owners manual for torque specs and check your rifle - when it heats up it causes a bind in the mating result in a shifting POI. An inch lb driver is required for this, so may need to take it to gunsmith. Bit ofa guessbut my savage ML which prior to me bedding it was a bit erratic without being torqued properly - though that was a syth stock version not wood.

If the poi remains low after it returns to ambient. Are you oiling the bore and not removing it prior to the intial firing? Excess oil causes excess pressure which willalter your POI (often higher).

If your POI returns on a cold barrel each time that would pretty much rule out a scope or mounting issue, usually these show erratic POI's not a pattern such as you describe.

Trying a totally different brand of ammo wouldn't be bad either. I'd be inclined to try a 180gr offering a notable performer with regards to the 06.