First off, it sounds like you could use some game cameras out there. Then you'll know if all you have are spikes. The big guys may just be nocturnal or in thicker cover. I believe trail cams get more hunters to pass on small bucks more than anything else. Knowing what's out there raises the bar for me. The point rule is completely ignorant! Many bucks with good potential can have 7 or more points by their second year, but can be pencil thin or not as wide. Does someone actually think shooting them is going to help grow big racks? You should also be allowed to take out bad genetics. That doesn't mean spikes, but say deer with funky racks or no brow tines. Hopefully whoever is making your club's rules will get their head out of the sand.
Mike