The .54 caliber is almost 1/4th heavier then the .50 caliber. 177 VS 223 grain. That means a lot more down range Whomp when you need it. For instance, I'd hunt elk with a .54 caliber (prefer my .58) but I would be a little cautious with a .50 caliber. For deer the .50 will work just fine. It takes deer all the time. Its just the .54 down range carries more energy when it hits, and that helps plant deer.
I shot a deer with my Lyman .54 caliber Trade Rifle flintlock at 52 yards. She was facing me. I shot her when her neck was down, feeding. The ball went through the neck, through the brisket, the lungs, exited, broke her back leg, and kept going into the woods. She dropped where she stood. I do believe the .50 would have down the same thing. But that .54 is just a great shooting ball caliber.
I have .45, .50, .54, and .58 (sold the .62) and I am proud to say, I can keep track of my .... well you know. The .58 caliber 284 grains are even more dramatic when they are shot. They really ring a steel target.