I understand your concern & I used to have a similar fear. I have harvested a first year deer mistakenly before and I was not thrilled, but I also knew there was no way to prevent it.

I never knowingly shoot a younger animal, but it will happen time to time. I guess you have to make the call for yourself. Some folks care about it & others do not. I personally do care about that & therefore I take time to asses a deer before releasing on the animal. Just my .02

Best of luck this season. I hope you get a chance to hunt with some family - that is what its all about right there.