Well first off Welcome to the site
Now on your question, Do you use scent control? My best guess is that the deer are coming up behind you and gettin a sniff of what you are, and thats whats scaring them away. I would suggest Scent-Away Shampoo, Deoderent, and Dirt Scented Spray. All those can be bought at Wal-Mart.
Also, Do you call? A simple grunt may be all it takes to lure them into the clearing. I would suggest trying the Primos HardWood Grunter. It is my go to grunt, I love it.
Hope this helps.