Shawn,
You need a better scent control strategy!!! And when I say that, I mean a REALLY DEDICATED stategy. I was ALL of my clothes in scent-free detergent, use only scent-free softener and dryer sheets, keep my boots in the box they were bought in, and put them on and take them off at my hunting site. I ALWAYS wear a facemask (lt. wt. neck gaiter) that at a minimum covers my mouth, and the back of my head and ears, and I spray off with a little autumn blend cover scent that really does tend ot smell llike the woods. I wear scent-suppressing base layer under my hunting clothses, and I usually use a little buck urine for my cover. Look over your scent control strategy, and modify what you can, and then re-assess. I HONESTLY pay little attention to the wind, and I get near a LOT of deer.