The jury is out, however there are some studies that show that even a yearling spike has bad genetics and may need to be culled.
MOST yearling bucks will actually be 6 pointers and genetically will generally grow to have larger antlers than a yearling spike will over the rest of their lifetime. (Based on actual studies, not antectodal observations).
Here's a quote from a study. (I added the bold emphasis)
In all cases, the fork group is greater than the spike group, and ranges from 6.7%-15.2% advantage for main beam spread; 13.1%-54.4% for main beam length; 13.5%-22.4% for basal circumference; 12.9%-108.3% for total points; 31.3%-125.6% for total antler weight; and 10.2%-15.3% for live body weight. It is difficult to understand how this data can be interpreted to prove that there is no difference between spike and fork-sired deer on the one hand and to state that only 20% of the spikes are inferior and 80% "will likely develop into decent bucks if allowed to grow to maturity." If you will review the tables carefully, they will reveal that in almost all cases, the fork group was equal or better at 3.5 years than the spike group at 4.5 years.
Here's a summary of one of the studies -
http://www.tpwd.state.tx.us/landwater/land/wildlife_management/deer/spikes_not_inferior/