Maybe the stands are on your side of the tree to keep the tree between him and the deer. Also sounds like he might be hunting with his son or a camera man. Not that that has any bearing. I'm sure once they see you constantly walking the fence line to your stand they'll find another spot.