IMHO it depends on the gun design and I can speak only for the Remington 700ML (sold it to my friend) and current SS Omega with camo thumbhole stock.
I use to remove Remington’s barrel only after the end of hunting season to give an overall cleaning. Did not see any change of POI next year.
It is more difficult to clean the trigger group on Omega, so I remove the barrel after every trip on the range, but I use adjustable torque screwdriver ($60.00 invested) with 25 in/lb settings to tight stock screws. No change of POI at all.