I too own land in Northern Wisconsin. I live here, I pay the taxes, and manage my land. I keep my trails passable for easy removal of game and enjoyment of my land for walks and bird hunting. And I will shoot anything legal, that I want to shoot. I have no fence neighbors unless you count the federal forest. But if I did, it would make no difference. I would still do the same thing.
My theory on monster bucks.. if one happens by, lucky me. If a four pointer passes by... lucky me. If you want to raise monster bucks or hunt only monster bucks.. good for you. Maybe I can have one walk past me, or then maybe shoot his younger sibling.
If mr.mc54 wants to shoot small bucks all day long on his property. Good for him. I'd back him 100%. Even if the neighbors want to practice buck management measures. They do not own the deer herd. And it is not for anyone to say what someone should do on their own land.
It sounds like your neighbor has a real mental problem. I would leave him alone. If I saw him near the fence area I would have a talk with him and ask him why he pours stuff on the ground or ties garbage to trees. If he is not willing to explain his reasons to you, then he can stuff it. But I would photograph him doing his stuff and his garbage and the first time one of his bags blew over on my property I would be on the phone to him. If that brought no satisfaction, then it would be the sheriff''s department.
You hunt as long as you like and any legal way you like. Good luck to you. I hope you shoot a nice deer this year.