Don't worry about it. I do it all the time. Let the little ones go and if you need the meat, shoot the does. I have two does in the frezzer this year, and donated another one. I feel better about that then I would have shooting a small buck, which I could have done on numerous times this year.