Got to get you a trail camera. It will unravel the puzzle for you real quick.
Sounds to me like the deer are moving through at night or you could be spooking them in the mornings going to your stand.