The only two factors that humans can control ina wild deer heard is nutrition and age. Culling a 4.5 year old deer because his antlers are not a certain size accomplished absolutely nothing for the quality of your deer heard.
It keeps that buck from passing along his genetics for years 4, 5, 6, 7, 8, 9 etc. which equates to many many does and yearling that aren't running around with his genetics.