I have the same problem and no good answer. The problem is the opposite in the morning - I spook the deer in the field on the way to my stand.

Have you asked the neighbors for permission to walk across their property on the way to your stand? Let them know you won't be hunting, just walking through.