Have you thought aboutputting up a couple Trail Cameras?
You could camo/hide the real one and put up a couple fake ones that are out in the open for them to see...facing right at thier stands!

I would probably talk to them and see what their real agenda is? If it wasn't on the up and up, I would definitely try to change the pattern/trails of the deer, probably not a fence. But, cut down some unwanted brush/trees and pile them up on the fence line.

You could be hurting your hunting also. I would plant food crops on the opposite side of your property so the deer would have to penetrate your land a little deeper to get to the goodies.

Years ago I had the same problem with a land owner who kicked a tree stand out of a tree I was huntin' opening morning for gun season. It was about 15 feet off the property line and I had just placed it in the night before. I just left it there and placed a portable about 75 yards deeper in the woods for the second day of the hunt.

Guess who I caught tresspassing the next morning? I was in that very stand an hour in a half before first day light so I wouldn't "push" them any deer. Twenty minutes before first light, the son walks over to the point where my tree stand was located ... lights a cigarette and puffs away, oblivious to my presence and sits down with his back to me.

My blood began to boil ... as I began to climb down the tree and confront this jerk! Every year after that during gun season, I antiscipated how they would push deer to me and began to utilize this information to harvest quality bucks. The key was to arrive "at least" an hour and a half before day light ... it payed off handsomely.

As they say, "when someone hands you lemons, make lemonade.