Antler growth is started every year anew.
When I lived in Pennsylvania we had a deer farmer, he bred deer for meat for restaurants.
No deer there ever lived past 2 years, but some of the racks that they had from good feeding, diet would make you swear you were in buck heaven.
There are no genetics in antler growth, they have shown this time and time again.
It is like you growing your fingernails.