Yes, it does matter. You are right and your friends are wrong.
Coming into bowhunting, I know I underestimated their nose. I don't, now. I agree with Greg, wholeheartedly.
People can say anything they like....and believe such. A hunter can't mask ALL of his scent......and the exposed areas are (IMO) the WORST for transmitting human odor (the mouth being No.1).
What would bother the crap outta me is.......wondering what smelled me and skirted me....that I never heard nor saw. We've all been busted by our scent. Period. If I get busted, opening morning (or any time after), I'll find solace in knowing it was my setup (which allowed deer to come in from downwind of me) that did me in....and not my refusal to properly play the wind.
There are plenty of instances where you can have deer come in from downwind of you and not smell you. But, these have more to do with currents/thermals.....and nothing (IMO) to do with them smelling a "little" of you....and not caring.