You say you agreed to let him hunt. But did you say when or what he could hunt? Seems reasonable to me if you let him hunt maybe a couple of weekends and limit him to taking a doe or two if legal if you don't want him shooting off the bucks. It's your land so you can make up the rules. Also tell him to check in at the house when he comes and goes and that you want to see any deer taken off the property. This will make it hard for him to sneak out one of the bucks.