On our farm we have a very good relationship with our neighbors. We allow them to pursue deer on our land, and they return the favor with us. We have even had at times let some cross our fence to hang a stand as long as it was not interferring with someone on us. While I don't bowhunt on any neighbors, I do have a very good shotgun stand about 100 yards across the fence, which the landowner has approved.

So I would shoot the deer with no fear of going and getting it.