Hammer, I started letting small bucks (spikes, 4 points) walk during the 2002 season. On the second day of gun season in 2002, I saw a decent-sized spike which presented me with several good shots. Normally, I would have taken him but I passed him up and about 30 minutes later, I ended up killing my best deer so far. Passing up deer improves your chances of getting something better, that's for sure.
For where I hunt, he's a trophy. Here's a pic of him: