I have two buddies that do the same barrel swapping as your talking about, without any problems.
What do you think recoil does to your gun. Do you think it might be comparable to tapping it with a hammer?

I have another buddy that have had two Winchester model70 rifles that couldn't hold a 5 inch group @ 100 hundred yards.
Sometimes you get a bad gun??