I suggest you talk to him. Maybe he has bad intentions . . . maybe he doesn't. You won't know unless you talk to him. He may lie to you, but if nothing else talking with him will put him on notice that (1) you care, (2) you know what his face looks like and maybe (ask about this) who he is and where he lives, and (3) get it clear what the expectations are.

Maybe he thinks the deer are moving from your grandpa's property onto his property and wants to shoot them as they cross? I don't know if this is a plausible story, not being familiar with the lay of the land.

Talk to the guy next time. If he is a jerk when you talk to him, that would be a good grounds for trying to do something else to thwart his plans -- maybe the Irish Spring on the fenceline?

Maybe the guy is a poacher and the adjoining landowner would appreciate some identification of the guy. Get the guy's license plate number and a good physical description.