Many different subspecies of whitetails exist in North America, so when comparing a buck from Sascatchewan (sp?) to a buck in Florida, the one from Sascatchewan obviously has better genetics for producing a bigger body and a bigger rack. With that considered, the single most important factor which determines antler size in any given geographic area is AGE. Age influences antler growth/size more than genetics, food sources, or stress. For example, a "dink" spike buck that is born very late in the year usually catches up to his peers and sports an equivalent (if not bigger) rack by the time he reaches 3 years of age. This would support the fact that genetics have very little to do with a buck's size. That buck possessed the same genetics when he was a year old that he had when he was 3 or 4 years of age... Genetics don't change over time.
Every credible source of information that I have read supports this thought.