Your rifle will determine the powder charge. You need to get to the range and try the different projectiles at different powder charges. The max of your rifle is 120 grains of 2f. Do not go over that.

Most of the time your projectile will shoot the best possible group between 80-100 grains of powder in a .54 caliber. Any of those powder charges are more then enough to kill a mule deer.

Using 80 grains of Pyrodex and a .54 caliber patched roundball, I shot through one deer and killed a second deer standing behind the first. A dump mistake on my part, granted. I had tags to take care of the incident but it was not intentional, just one of those things. The deer were 80 yards away. That might indicate to you the power of a roundball. By the way, both deer dropped in their tracks.

What chance do you think a mule deer would stand at 80 yards with that same load headed at it?