I do not know this for a fact, but I have been told that in some states you have to be a certain distance from the property line, and you can not be facing the property line. This keeps someone from hunting a property they don't have permission to hunt. Look into that. But also be civil. Let him know that you do not want him hunting your land, but you would gladly let him track an animal on your land, and even help if he needs it, in return for him allowing you the same courtesy.

Just try to calmly address the situation first. And it might be better if your grandpa was there when you talked to him, being the landowner. Just a thought.