If the deer is facing away from you and you shoot it in the A$$, YES the 454 will kill faster providing you can handle the 454.
I have a Ruger Super Redhawk in 454, I can shoot much better groups with it using 45C ammo than the harder recoiling 454 ammo. With practice I am improving and I find a good pair of Uncle Mikes shooting gloves as well as aftermarket grips helps.