If your scaring deer out in the morning on your way in chances are there is a bedding area not too far. Study the trails to find the bedding areas and the staging areas. If your area is real close to the bedding area you may have to get there a couple hours before sunrise. If the bedding area is far enough away to allow you to set up between the bedding area and the feeding (acorns) area try setting up there and always keep the wind in your face when you enter the woods. Do you have options on which direction you enter the woods from? It sounds like it may be a good afternoon stand?
Keep the wind in your face!