That's the tough thing about hunting public ground, there really isn't much you can do except move yourself. If you confront them, you never know how they'll react.

My suggestion would be to do what you can to move further back in the woods. In my experience, most people prefer to stick close to the roads and don't want to hike any real distance. I usually get 1.5 - 2 miles back into the woods if I'm hunting public ground.