RE: does anyone worry about
I wouldn't worry about if you are doing it now and are done hunting. I'm sure unless you are hunting is some unknown pristine area, that the deer will be bumped many times from now till next season.
Try and get in when they'll be else where but I wouldn't panic if ya bump one!