I have a set up just like yours only it's on a Remington 1100 with a Leopold scope. If I were you I would always re-checkyour zero after putting on the barrel. Not because the scope will be out of adjustment. If it's a quality scope it should hold zero and being mounted to the barrel via a cantilever it should be rock solid.

What does move....and what some have already posted is the fit between the barrel and the receiver. Take ANY Remington shotgun 1100 or 870 and you will be able to move the barrel once it's been installed. There will always be some play between the two, it's just the way it is. Some are tight enough to where you will get a reliable zero time after time, some will need to be rezeroed.......and some, well some won't hold zero from shot to shot, there is just too much play between the barrel and the receiver, especially if your slug barrel is an aftermarket type like hastings etc...Don't get me wrong they are good barrels but the fit is not always perfect.

So, re-check your zero after you put your slug barrel back on, it may be fine.....it may not.

That being said, I don't hunt with that gun. It tends to hold it's zero but I find my Ultra Slug hunter is far more accurate a slug gun, even my Mossberg 695 will out shoot it. Both are far better "slug gun" platforms than the Remingtons especially at longer distances.