I think you are overreacting unless the area has a lot of pressure. Even then, it does not matter. Why is that? Because other people affect their movement which would cause them not to leave, but to become human wise and nocturnal.

Deer don't leave an area permanently unless they are forced to. They just leave, and then come back. Let's face it, if they did not return, hunting would be far more challenging.

I walk through bedding areas scouting every year. As do other members in my camp. We see, and more often than not, kill deer every year.

Stop worrying, go scouting, and be successful.