I'm not surprised the Hornadys don't group all that great with your 500. My gun hated them. It prefers slugs in the 1350-1550 fps range. Try something like Winchester Super X 3" Sabot or Brenneke KO (the true sabot version, not the attached wad one) and I'll bet you'll see smaller groups. Be sure to tighten the barrel nut firmly, and don't rest gun on the forearm for sighting in, rest it on the receiver.

I have a Marlin 512 and though it is not much to look at, it is a great 100-125 yard deer killer. That's enough range for me and I shoot the "cheaper" sabots. The Slug Warrior is a great gun but can be pricey. Another bolt action is the Mossberg 695 (I think). The Marlin and Mossberg are not made these days, so you'll have to find one used, which can be done. They are making the Slug Warrior now.