I have had similar experiences before I suited up. I've been undetected and busted either way. The wind does strange things and your scent follows along I guess. One thought is that if the day is warm the air has lift and just maybe your smell went over top of the deer? I used to have some of those "floaters" which you could follow in the air currents and see where you smell was actually going. The puffer type lets you know what is going on around near you but fades to nothing in short order. Once I was setup along a creek bottom in the hill country of Texas and my floater went several yards down stream as expected and then turned uphill and circled almost behind me. Tough hunting a swirlling wind!