Yeah, if the deer is close enough to spot your glasses he is probably so close that you have either already taken the shot or decided to pass.

I wouldn't worry about it.