I had this discussion with a DNR rep recently. He said:

Young doe(<1.5 yrs old) are the best deer to remove from the herd. The older does' fawn have higher survival rates because they are better mothers - more robust mammaries, more predator savvy etc.

The trick is that it takes disciplined shooters to make sure that there are no buttons on the small 'does' that are being shot.