I've spooked them, oftentimes it is an odd sound, something not of the forest. Like resting your rifle on wood, that clunk will often spook them. I put indoor outdoor carpet on the rails of all my shooting towers. Hitting your rifle butt on something will do it. I've spooled them just by putting my safety from safe to fire. All it takes sometimes is any odd sound not common to the forest.
One thing I've watched other hunters do is to try to look in different directions quickly. Whipping your head around not only spooks them with the motion, but the flash of your pale face also stands out even at night.
Not much I can do about my scent even using precautions, you have to breathe and Deer can smell parts per billion. I move to my shooting stand really slow and as quiet as possible.
My assumption is often Deer know you are around within a 3-400 yard circle. Not only their senses are attuned, but they also notice when a bird sounds the alarm and other subtle things, they are attuned to their surroundings. IMO most of it is if they consider you a threat or not. If they spooked at every event, a leaf rustling, a twig falling from a tree, grass brushing together they would be running all the time. I figure their noses are at least as sensitive as my dogs and he can smell a Fox at four hundred yards.
I once fell asleep in a shooting tower and when I woke up there were fourteen Deer standing 30-40 yards in front of me. I'd been in that tower for hours and hadn't seen squat, obviously, when I was active they noticed, when I was inactive they relaxed. Learning to sit still and not fidget is a big part of it.