Let it be. Report to landowner or warden if thee is a second encounter .... or if you find clear signs that trespassing is going on.

My suspicion is that this is not the first time nor will it be the last time this guy trespass hunts on that property. He'll just be looking closer for you now before "crossing the fence".