Dude, he is a man just like you. Don't play games, he will see thru them and in the end not respect you. He doesn't respect you now.

Go up to his door and in a gentle but firm way let him know you saw him. Of course he isn't wearing orange, he's trespassing. Look him in the eye and convey to him you do not want him there and if he had asked in the first place it might have been different.

If he denies it, fine. Let him know that if you see him again the authorities will be called. A guy who has done this once shame on him. You get my drift.

Good luck.