Deer are crespuscular (meaning that they are most active at dawn and dusk).

When I gun hunt on public land where I know guys are going in for coffee or for lunch, I stay in my stand all day long. (most of my success has come around 9:30AM or 10AM)

During the rut period, I stay in my stand all day long.

At other times, I usually hunt until 10AM or until I get tired and then go back in at 3PM until dark.

I guess my opinion would be to stay out as long as you can, because you just never know.