IF you can seepatches of leading in your bore that obscure the rifling in places, then that is definitely what's causing your slugs to fly all over the place. Such leading will have to be completely removed down to the bottom of the rifling grooves before you can count on the slugs getting a sufficient spin imparted to them to start giving you acceptable performance.

As mentioned above, use a GOOD lead-removing solvent, and you may have to resort to using steel wool wrapped around a bore brush to cut thelead out, if it is real deep.