I wouldn't worry to much about shooting a button buck.

The bucks that really need to be spared if your managing for big bucks is those nice 2 year old bucks. People shooting those is where it really starts effecting the heard IMO.