While I would've done my best to avoid a heated exchange, I would've certainly confronted the guy either in the field, at his vehicle, or somewhee in between. I most certainly would've called the authorities too. Oh well...maybe he'll stay away now.
I had a guy on private a few years back that was in my lock on stand when I came in to hunt one morning before sunup. He tried to tell me he thought he was in his stand. I mentioned you're on private property and he said "Really?" and gave me the whole line of BS about being lost. I hate being tied to a cell phone with my job, but I never go hunting without one now.

Whoever mentioned knocking the guy on his ass...That's a good way to get shot, and if the authorities were contacted you'd get a bigger fine and sentence for assaulting the guy than the trespasser would get. That's showing 'em...eh?