I don't think nature is taking care of the problem. Just because they aren't all bredding doesn't mean that too many aren't breeding (I hope that makes any sense). You may have a higher fawn mortality due to poor nutrition, but because of your temperate climate, the deer will not have near the mortality rates they would say in MN, so they will continue to live and continue to be smaller than they otherwise would. I am speaking of body size here not rack size. I bet if you could some how whack 50% of your doe population, you would see a sizeable increase in average body mass over the following couple of years.

Or you could kill all but a buck or two, but I would try the other option first