I just look at body characteristics and behavior of the deer. An older deer is more cautious and aware of his surroundings and where is in relation to them at all times. Younger deer tend to be a little more carefree. Here are some pics of deer on my place.
2.5 year old deer. Smaller neck and narrower shoulders and slender snout.
3.5 year old deer. Bigger thicker body and larger antlers. Bigger head and shoulders.
4.5 year old deer. Real big head and thick neck. Thick set of antlers. and big heavy body.
Another 4.5 year old deer. Big body. Big neck. Sway back. Big head.
Another buck on the same scrape as the buck above but this is a 3.5 year old deer. Notice the body structure difference.