What you might try.
Since it sounds like you might have some sharp edged pits which will disrupt accuracy I would either flatten several lead balls of 50 caliber or take one of 54 caliber and force it through the bore take value grinding compound and lap it taking 5 strokes and adding more compound until you have reduce the sharp edges on the pits. If its to bad you might want to replace the barrel.
If you do not have any sharp edge pits and you think it is just the load try the Hornady FPB 350 gr with 80 and 90 gr Pyrodex RS and see if that does not shoot a bit better.
Be sure to check all your sight and mounting screws as you may have more than one problem.