The biggest rack buck Ive shot was also the smallest body buck Ive shot. The size of a deers antlers will not determine how hard he will be to kill but may determine the margin of error you are willing to have.

To put it in perspective a buddy who also happens to be a taxidermist, kills big bucks consistantly every year and his gun of choice is a 22hornet h&R. single shot but he rarely shoots more than 40 yds and doesnt get very rattled when he get a big buck in his scope.

Im not endorsing 22s for deer but a 100gr 243 and up is a legit 80 to 300lb deer load IMO if it goes in the right spot.