If you claim to be a true meat hunter then you already know that taking younger animals can mean (and almost always does mean) better tasting venison. I enjoy ALL aspects of hunting, and especially enjoy the care and treatment/preparation of the kill. My preference is to shoot 3-4 yr old BIG does from agricultural areas for the first deer in the freezer. If I am successful at that goal I can relax and be picky looking for some horns. But I've killed, butchered, and prepared enough venison to know that the big, burly, mature bucks taste nothing like a 1 1/2 yr old deer. I've been cooking a long time and I'm quite good at it. Have been doing most of the cooking in my home for the past 20 years and am often complemented on the food I prepare. Folks rarely turn down a dinner invitation at my house, and if they have to turn it down, they always sigh and complain about their other commitments, Doesn't happen much.
I have quite a few racks in the garage and basement, but have only paid to have one mount done. None of the racks in my basement, garage or on my wall have any flavor what so ever. Some of the flesh was ok, but the horns never made it to the pan/broiler/grill/stew-pot etc. The one deer that now graces my wall was field dressed immediately, kept cool, and aged for about 10 days. Still tasted like cedars and moss.
Point is, every deer you take can be a fine animal. Taking mature bucks or immature does and anything in between can only be measured in terms of the deer population in the given area.