You should invite him to hang a stand on your property and then maybe he'll move his bait pile on your place.

But seriously, the deer have got to come from somewhere to get to his all you can eat buffet. Chances are that some cross the fence to do it.