Scent isn't that big of a deal if you play the wind. But any advantage you can employ while in the field will help. You don't always know which way the animal is going to approach and more often than not, the mature deer tend to approach from down-wind. In that case, anything you can do to reduce your scent will help. You'll never, I repeat NEVER, be able to eliminate all of your scent. But you can do what you can to reduce it.

I saw a study one time that tested what factors cause scent to linger in a certain area. They showed that human scent will linger in a wet environment more than in a dry one. So the day you were at the feeder might have been dry and your scent went away before the deer arrived. Maybe not, you'll never really know what's going on until you can speak deer and ask them. All humans can do is make educated guesses based on research and trends.

I'd say movement, noise, and scent are the most important things to pay attention to when hunting. That's just my $0.02 though.