I would do my very best to avoid conflict. Nothing good is going to come from that. Since the man has already said you cannot cross lines to go after a downed deer, I would do my best to away from hunting the edges. However, with that being said, if I felt "the" spot I needed to be in was close to the line, I would hunt it in a heartbeat and just be prepared to get the law involved if one does cross.

As far as what I would have told this guy....I would have simply said thank you for calling, I will hunt this spot when and how I want since I have permission to do so. I would have left it at that.

On a side note, since it seems this guy already has a beef with you being there, make sure your I's are dotted and T's are crossed. Hunting on someone elses tags will get in you trouble real fast. Have the land owner give you the money so the tags are in your name. This guy sounds like he is going to be looking for a reason to get you out of there.