Sounds to me, according to what the poster said, that the deal was he helps cut trees and he gets to hunt. Unless you told him you wanted help on the rest, then you need to stick to your word. Four acres is not holding six bucks. So you may potentially lose them to neighbors as well. If you made the deal with them (Trees for hunting) then man up and stick to your word. If the deal was Trees and he help with bait etc, then you have a foot to stand on. But don't back out on your word. Count it as a lesson learned and don't invite him back. I don't know what kind of hunter he is, but even if you had six bucks coming in regularly, I would not be interested in hunting on just 4 acres. I like room to roam in gun season. So maybe he won't even show up. Good job on taking your first deer though, hopefully many more to come!
-Jake