Alot of good points have been brought up here. It seems to get back to the area you hunt, number of deer you are allowed to kill, deer population, and quality of the deer herd.
I live in a state with a long season, liberal doe limits, and a 3 buck limit. To enhance the herd, I try to fill my freezer with does, but I hunt for the big antlers. I usually take that doe when I'm about to retire for the morning or afternoon and nothing else is around. I guess that has driven off a few large bucks that were sneaking in, but I'd rather take a doe than a young buck. We don't have monster bucks but we do have 120 to 160 class bucks and that's a trophy to me. I wouldn't shoot a 6 point with the exception of that one Taz saw, a 20+ inch 6 point would be fair game with good mass.
Everybody is different, when I first started deer hunting, I'd shoot any buck that came by but in my old age, I'm happy to let the young ones grow.