Before you go sanding anything, or spending a tank of gas on ammo, try just putting a slight upward pressure on the barrel near the end of the stock. I know that free floating is the norm, and is usually desired. But if this is a hunting rifle and not a bench rest gun you're looking for 3-5 inch groups at the longest range you can keep such a gorup. Sub 1 inch at 100 yards is great but 1.5 " at 100 is pretty good. If your rilfe (and you) can shoot to 300 yards or beyond then your groups wil certainly open some at those longer ranges.
My point being that a small piece of thin credit card or other non porous material causing SLIGHT upward pressure on the barrel can sometimes stop the little difference seen when a barrel changes temps.
If it's a hunting rifle you're probably looking for getting of a couple shots at most at one time in the field anyway. It's a very cheap method to try when you believe your scope, rings, bases, and action screws are not the culprit. Costs next to nothing money wise and takes maybe 15 minutes accomplish and find out.