Been there, done that.

The first time I passed up the shot. He trotted away never to be seen again. No regrets on my part.

The second time, a buck was 3 yards from the base of my tree facing me. I didn' t shoot. He eventually offered a good angle and took a broadhead through the chest. 70 yards later he was down for good. Still no regrets.

So no, antler size does not dictate proper shot selection for me.