To many variable to say:

- You say he is "to close" to your areas. What if he doesn't think it's to close?

- why are they "your areas", he has a point he has just as much right to be there, there should be a way to work this out. Is he encroaching while you are there, or while you are not there?

He has permission, same as you. Yes there needs to be some conversations, but a third person has now joined you and your neighbors "agreement", one he didn't participate in. Time for a new agreement to be discussed.

As for age: I'm 50.