Yes, I agree a swirling wind is difficult.
It is also difficult for the deer especially if the noise level in the woods increases.

I always keep my hunting clothes in a big container away from other nasty odors.
I never use cover scents or scent killer sprays. IMO, you can never be scent free and using these products may be helpful, but it is not enough to fool a deer's nose.

I always try to hunt the wind.

I also make sure I NEVER touch any branches, twigs, leaves, etc with my bare hands.
Leaving nasty, oily, human scent in the woods is the worst thing you can do.
Even on warm/hot days,I wearthin gloves. Especially when putting on hunting clothes and boots.
Some hunters us all kinds of cover scents and scent killer sprays yet leave their scent on their boots and hunting clothes when they get dressed.

I also make sure to walk through any water to help the scent down from my boots.
My suggestion is to continue to hunt the wind, even on nasty days, and keep your movement to a minimum.
Good Luck!