It wouldn't be considered a "Pen" if the is a high fence on just 1 side.

I can certainly feel your pain. I haven't had you situation arise but I did plant a couple of small food plots this year (less then an acre total) so I know what kind of money, time, energy, and sweat go into them. Have you ever asked the guy for a hand or seen if he was interested in land management. Possibly he feels out of place coming over to help.(I don't know the situation)
If you want to keep deer from crossing the line my best suggestion is to give him about 15 feet of chain link fence on the entire property line. It'll probably start the feude you'd like to avoid but it'll also slow down the migration of deer off your property to his.