You really need to talk to your neighbor...

If he has given permission to other people, they have no idea your hunting the property, because you don't park your car at the house. Hence, they may think your a trespasser, especially if they have permission.

If this property isn't over 50 acres, preferrably over 100, I would just leave if I knew someone else was working there first. I'm not sure what your hunter density is per mile/property, but personally I've had 40 acre lots with 10 gobbling toms, and also have hunted 250 acre farms with only 1 dominant gobbler. Therefore, you can't fault a guy that was hunting a piece and "moved in" on your bird, if that was the only bird gobbling...

Don't always assume that it was the other guy that ruined your hunt...maybe you ruined his...