IMO the wind can disrupt the normal game movements, it can also draw out the really willy critters, the big bucks, or willy sounders. IMO a few things going on, deer and other wildlife need a lot of food in the normal course of things as their forage is generally poor quality, they make up for it in quantity. Sometimes they have to feed whether they want to or not. A light breeze spreads your scent a high wind dilutes your scent and it also covers your noise.
I was once hunting a Perruque-head Deer and the only time I ever saw it within a mile was in a sideways ice rainstorm, at about 30-40 yards. The downside was I was shivering so bad I had to pass up the shot.
Some critters are way smart and use inclimate weather to their advantage. You are right most times that high wind or pouring rain tends to make them stay home and hunting then isn't a high percentage proposition.
When I've gone after Red Deer I go out on the coldest day I can. Two reasons, the weekend hunters don't like the extreme cold either and the big one it is a good time to find Red Deer in the Spruce and Pine woods. If you get some standoff distance on a really cold dawn you can see the breath from the Red Deer going up through the needle forest like a small smokestack.

No wind today so I decided to hunt. Wish I stayed home because I hit a deer in the dark on the way to my spot. Full impact at 55 deployed air bag system. Lots of damage to my gmc.

Oh man, that is not the way you wanted to get your deer. Sorry to read that.

You all good CI? Air bags can do allot of damage to a person.

I hit one back in February. That'll ruin your day.

-Jake

Hope you're safe CI. You know the Ford guys are gonna tease you though.