I have gone out in 60 degree weather here in NY during the start of the season and seen deer moving when it was so warm you would swear it's a waste of time. In fact I got busted by a buck because I wasn't paying attention during weather like that.

I've also gone out on a crispy 32 degree morning in the peak of the rut and not seen a single thing all morning long. It really makes you scratch your head sometimes, but the most important thing is to just get out and put your hours in.
In my experience, hunting when it's really windy can be good if you hunt an area that is protected from the wind.