I do it all the time. I think it really all depends on what's important to
YOU. As long as you are proud of what you shoot, and that makes you happy....go for it. Personally, I don't want to shoot any more small bucks. I'd rather go without, and have done so plenty of times. As far as does are concerned, if you have a high population...I'd recommend whacking a few of them instead of the small bucks. I do pass up does as well, but that is based soley on whether or not I feel like dragging and butchering on that particular day.
I see you live in Madison, VA. If I lived down there...I'd be passing up every small buck that came by. That's one of the few areas in our state that has some real bruisers. Good luck