Theres a science to it, but you can definitely tell an older buck when you see one. The problem is you just don't get to see them often enough. Mostly due to the fact that A) we shoot most of them before they get above 3 1/2 years old. B) because when they get above that age they are very elusive and nocturnal.
check this article out.
http://www.qdma.com/articles/details.asp?id=76