I recall one year, I took a big bodied buck with a poor rack (DEC check point said he was 5 years old). I did this for two reasons, the primary reason was genetics, I didn't want him breeding does if I could help it. The second reason was for meat. I hunt private land that is not fenced but if I can do my share to help larger bucks pass their genetics on then I think that benefits everyone hunting in the area, including me.
It's nice to bag a big rack (IMO) but you also have to be willing to take does and bucks with poor genetics to get a big rack down the road. So if I get a chance at a big rack one year, I'd like to think the efforts I put in those previous years helped. If I don't bag one, I'm still happy because I'm doing what I think works for me.