Well I started with a 40 and tried every common caliber up to a 62, they all killed deer but for consistently dropping them on the spot none compared to a 54. You also give up the long range capabilities no other PRB compares well to the 54. Foe sabot-ed bullets is when the 50 shines the 58 is nice for a massive slow-moving hit on really big game like a moose but with a rainbow trajectory and a ball that is very heavy and a twist that is designed for loads that don't give massive recoil it also has some disadvantages.
For PRB stick with a 54 caliber other wise you give your self unwanted disadvantages.