I have been through this too many times. We lost a spot behind my house back in 2003 because too many people were bitching about sharing 200 acres, I think the major issue was that a 196" typical was taken in the area and a few of us saw one that was bigger in 2002 and everyone wanted to shoot it at any cost. The price was we all got kicked off and nobody hunts it anymore.
I have another spot that has neighbors poaching, I also have a few hunters that are cops hunting it and I told the land owner I was cool with them using my stands (8 on 50 acres). That area I rarely hunt and only go on weekday mornings.
My honey hole is a 5 acre parcel with another 15 next to it that we can hunt. That area has a guy and his son that have hunted it for eight years, me for five years, another father and son for 3 years and another guy for the last two years. I hunt this property hard and take bucks every year, we have had a few ups and downs and the propery is small but a little property is better than none.
My best advice is to see if the guy will go out to lunch and request the landowner to go or see if the three of you can get a 12 pack and shoot the bull. Advise him of your concerns and try to develop an understanding of communication so you can both hunt and not be on top of each other. Offer to share stands and work together, convince him that two people working together will be more productive than two fighting. Not only have I saved my current honey hole but I have also gotten a patio for the house done much cheaper through a fellow hunter, get my driveway sealed for free and have gained permission at to hunt with them on other pieces they hunt! Good luck and try to work with them