Often times a slug gun with the scope mounted to the receiver will have issues like this due to the way the barrel mounts to the gun.

As mentioned above, when the gun recoils the barrel moves in relation the scope, causing your patterns to open up. Companies such as Tar-Hunt which produce some of the most accurate slug guns out there pin the barrel to the receiver to stop this independent movement.

Another possibility is that you are shooting too fast. You need to allow the barrel to cool between shots. The walls of a shotgun barrel are fairly thin and the heat can cause your gun to start throwing flyers. Granted 6" at 50 yards is pretty significant and probably not 100% heat related, but it can certainly contribute. The most I take is 2 shots before allowing the barrel several minutes to cool back to ambient temp.

One other possibility (although it would require a diferent barrel) is to use a barrel with a cantilever scope mount. In this case, the scope mount is welded directly onto the barrel. The scope and barrel come off as a single piece. Unless your scope is trashed or your mounts are loose, the gun should hold zero almost indefinitely (Mine has been zeroed for at least 3 years and has been on and off the gun repeatedly)