RE: Shouldn't they be bigger?
Cull them. They may never grow out of being a spike and will certainly never mature into a trophy.
I've never read or seen a scenario where eliminating spikes was a mistake. I have seen several that had a policy of leaving spikes, and on each of those, it was clearly a mistake 3-4 years down the road.