I think the guys have this one pegged. Hunt deer not bait. The deer use the bait at night when people aren't around. They avoid your area because you or someone frequent it. The scent thing is very important, but even if you did way way way less scent control you will still get plenty of deer inside of 20 yards and they'll never know you're there.

Good Luck!