He's legally on his property? Who cares.
Just because he's near your land, facing it, doesn't mean he's hunting your land, perhaps deer funnel from your land to his in that spot, and he's there. Can't argue with his logic there. If he's shooting deer on your land, etc...that's a completely different issue.