Thats me too, I am a trophy hunter and a meat hunter, lets face it bucks are not as good eating as a yearling or 2 year old doe. So I pass on young bucks as well, if it's not something to be proud of I will let it walk, this year I passed on a nice 2 1/2 yr old 8 pointer, I shot a yearling doe right beside him, this was with a bow so it's still a descent trophy to me and the best eating you can find.
But I will not condemn anyone who shoots small bucks, if it makes you proud then take it, and thats the key ( if it makes you proud). Most hunters get to a stage in their life when they start letting young bucks walk, usually after years of hunting.