It really depends like what is said above, on where you are in relation to where your deer are. It takes time for your deer to get from point A to point B and many times that might not be at first light. In my opinion you moved too soon as well.

In some of my stand locations I don't see deer until midmorning. One particular spot we didn't even expect to see deer until around 9:00 am. Sure we'd be there at first light just in case but it took time for the deer to come from the neighboring Ag fields to where we were on the oak flats.