The following is from a wildlife biologist friend of mine:

"From a biology viewpoint, maturity in a deer is sexual maturity. Thus, half of our doe fawns are mature at 6 months. I really don’t think any button bucks breed, so their maturity most likely occurs at 1 ½ years old.

As hunters, we’re more interested in when they reach their peak body weight and antler growth. This is not a science question. I think it’s generally believed that deer have their best antlers somewhere in the 3.5 to 5.5 year old range.

We just arbitrarily say they are mature at that age."

That pretty much sums it up for me. We are arguing when they are their "biggest" not when they are actually mature.