The deer don't leave because it's cold or raining or snowing, so why should you. I shot my last buck on a cold, wet and foggy day when he strolled by 25 yards away.I've had my shots when it was-5 deg and during practicle whiteouts with snow. Depending on the type of weather,it may make them move even more than normal.
(Some of the best hunting for my father, brother and I has been when the weather turns crappy and all of the other hunters nearby pack up and head for home. Get's what is out there up and moving, pushing deer all over, and right into our sights)