One thing funny about round balls, due to their shape, the faster you push them, the faster they slow down...Add to that the inefficiency of black powder where it reaches a place where more powder simply doesn't give much more velocity and you find that a charge of 70-80 grs of powder is fine for a .45-.54 round ball...
And, let's not even get into burning patches, which more powder will do...
And more powder also leaves more crud in the barrel...
When I finished my .54 I started with 120grs FF, after a few years I went to 100grs FFF and then down to 80grs FFF...I suspect that 70grs would be plenty but with a 9 pound rifle and a 38 inch barrel there just isn't any real felt recoil so I figure I'll leave it alone...
Another thought as well, a pure lead ball sure flattens pretty easily even at slower velocities...I've seen them flatten to about the thickness of a nickel...This retards penetration...So it's really not what you want to see...