Every choice you make controls the herd. This is the same argument as "I don't vote because it doesn't make a difference" All those votes together make a difference. Same thing here. No matter if you have 30 bucks or 3000.
I may not see a legal buck for several years. I'm not making a huge impact on several thousand acres. Seriously, there are about 6 bucks total killed on these 3500 acres a year. If this was heavily hunted SGL and multiple hunters got chances at multiple young bucks every year, then yes, I would agree. 90% of our bucks die of old age no matter what we shoot. Not enough hunters and a low hunter success rate for those few hunters means very low impact on herd.