By shooting a little buck, and not a doe you are ensuring a future season. A buck is easily replaced, as a doe is not, and can have one or more fawns for next year. I never shoot does, and only shoot a small buck if its the last day of the season.
And as for " trophy only" hunting; its a matter of personal preferance. I myself consider myself a trophy hunter, but will take meat for on the table.
I have heard the " you can' t eat the antlers" phrase quite a bit, but the bigger the antlers --- the more meat there is under them