Some basics:

Year 0 - Buck is born in spring and is a "fawn" buck in fall with only nubs.
Year 1 - 1.5 year old "yearling" buck has only "spikes" or "forks" or small racks. (EDITED - BarnesX is correct!)
Year 2 - 2.5 year old buck can have a full rack, but not heavy.
Year 3 - 3.5 year old buck has a fuller rack, but not as heavy as it could be.
Year 4 - 4.5 year old buck has a rack that is full and heavy, considered "mature" by most.
Years 5-7 - with good feed bucks put on even more mass in the rack.
Years 8+ - bucks start to lose mass in racks.

Genes and food source have a role in all this. This is what I see up north, where food is limited in winter and deer are stressed.

This is not the norm. We see plenty of 1.5yo bucks that are forkies, 6 points and even basket-rack 8-points. Spikes are pretty rare unless nutrition is poor or range is over-crowded.

Genetics and diet have everything to do with it.the world record is only 3.5 years old

Also hunting pressure and stress on a deer have an effect

Genetics and diet are true players. Thats also why even i admit when there is a drought you can see the effects of poor dieting in the deer. Growth isnt as good as it is in the banner years. Age also plays a role yet probably the least in the three factors. But how he lived will eventually increase its influence as time progress's. Got to keep in mind the older the deer gets the more worn his body becomes. The more nutrients his body requires for energy, possible wound healing, and other factors. All that just to keep up and ahead of predators, dominance, and alive. At a certain point the body will primary take the nutrients to where the body needs it and not so much where the rack comes into play.

Cuz unless you have a fenced 1000+ acre plot...the guy down the street/road whatever from you isn't going to wait to let the buck mature...as far as Im concerned you need to keep the GOOD bucks around...but its really luck of the draw because what you pass on may be the biggest buck the next guy has ever shot or has seen or just wants to kill it...its just a timing thing...IMO. quality DOES are also needed...Last year I saw a ragged antler spike with ears chewed off/half missing...I never saw it again...dunno if it got killed or just moved on. This year I had a huge 8pt on my property...and I haven't seen it yet...I cant shoot it because I already filled my doe tag but it would be nice to keep him around! We'll see how it goes.