Some of them old horse pistols were a night mare for accuracy. I actually in my life sold a black powder gun.. it a was .45 caliber Jukar kentucky pistol. It went off fine, but I would have been more of a danger if I had thrown the pistol at the target instead of shoot. I never was able to get any kind of a group with that thing and the truth be told, I used to be a very good pistola and revolver shot.

Try dropping the powder charge down to 25-30 grains of 3f powder. Sometimes you push a pistol too hard the ball skips the rifling and you just spray all over. Then try a patched roundball... Start your shooting at 15 yards and get your group down there, then move back.