Managing a deer herd is much more than "just letting the small bucks go". Here are some rules of thumb, not in any particular order.

1. Get the does in check. I like 3to1.
2. Only harvest mature bucks, regardless of antler size(for our camp, we classify 3 1/2 as mature)
3. The harvest of older cull bucks is a must and VERY important for the future buck herd.
4. Provide plenty of food and cover, year-round.
5. HAVE FUN