I would leave a civil note on the tree asking him to remove it and if it is still there in a few weeks I would remove it and set it just off my property in front of a posted sign with another note asking him to ask permission next time.

The last thing you want to do is make an enemy. They can make your hunting miserable.