Of my last 4 muzzleloaders, 3 of them could be removed from the stock and return to zero just fine; the 4th could not. The POI would shift as much as 3-4in at 100 yds. That particular gun was floated, pillars installed and bedded. Problem solved and now very accurate.

Chances are you will be ok, especially if you do some of the tricks others describe above, but there is the potential to affect zero if something about the barrel-stock contact changes. I'd at least try it once and shoot it to confirm it is ok before I did it during hunting season.