It all depends on the make of rifle. A pinned barrel I never remove. A wedged barrel, why not. And one held in by lug locking screws I always do. If I put that barrelback in the stock and the rifle shifts POA then I am guessing that a bedding job needs to be done. I have been removing them for years I never noticed a problem.

It just makes the barrels and parts easier for me to clean the way I do. All scopes are with Warne QD rings, so the scopes come right off. Then after I get the barrel and parts away from the stock I can use a water bath to get the main filth off everything and then just a few solution swabs, and then some dry ones and I am ready to oil the rifle up, check all the parts including the trigger assemble for fowling build up (which is kind of hard if you do not pull the stock.), and then get everything oiled, wiped down and back together.