I have never passed on a deer then failed to get a bigger one. I have, however, shot a smaller one then not been able to take an opportunity at a bigger one as a result. In MI where I spent most of my years hunting deer there is a two-buck rule, so few big ones around. Very often I shot a smallish one then kept going out hoping for Mr. Big but it never happened that way.
Here in IN we only get one buck, so I'm sure I will be forced to make a tough choice one year soon. There is a Booner or two on the property I hunt but I only get about 2 days a year there. This year I thought I'd only get opening morning so I shot a pretty runty one. Then I unexpectedly got another morning out later in the season and had to pass on a really nice 8-pt. Oops.