As I've gotten older I've given up on shoting deer with anything bigger than a 7mm projectile. I wrote a line or two on another thread about 7mm-08 and the 280 Remington. During our regular season I killed all but one deer with a .243. The 280 was used on a bruiser of a buck that weighed about 175 pounds and I definitely didn't want running off to get lost or tagged by someone else.
You are correct in your assessment, caliber and bullet weight are entirely secondary to a well placed shot. That's why we shoot so much during the year.