Two things you can do for next year since this years history. If there feeding, make small food plots. Nothing fancy, even no plow placed in heavily traveled areas will give you enough confidence they're gonna frequent your area besides his.
Second, if they're habits have changed, where do they cross your property, heavily traveled runways, place a camera and see if it's day or night, morning or afternoon. If they bed down in yours, could be to your advantage. Rather than fight him cause he's gonna do it no matter what, then use it to your advantage, know where they end up and just find out how they get there and when they leave. And besides, if it's amiable now with knowing you can cross lines to find your deer, is it something you really want to lose ? Keep the lines of communication open.