Hunting over bait can, if done incorrectly, lead to greatly decreased success. Like has been said before, the deer are patterning you and understanding when (daytime) you'll be out. You may want to try giving up on the bait for a while and just hunting regularly and see how that works for you.