Before I would go and change anything as both those loads shoot good I would try a couple things. Next time out take another rifle along. Shoot this one then play with the other until you're good and sure this rifle has cooled down. Then fire another group and see where it hits. Should be the same as the first group. If not, give the barrel a good cleaning there at the range and try shooting again.
IMO its one of a couple issues: Either heating of the barrel or barrel fouling. (If its barrel heating make sure your barrel is free floating) It may also be that the action screws are not tight.
Maybe a bedding job is in order but try the simple things first.