The deer are in a nocturnal cycle. This is due to pressure from hunters, predators, or even you walking in the area. So the answer might be for you to stay out of the area. Also, having the feeder on the edge of their range rather than within it would help. If possible, a food plot that you could leave alone for weeks at a time would serve you better.

I also have a theory that as deer are hunted over bait, the deer that are not harvested tend to be more nocturnal. We may be unintentionally creating a new strain of deer that is, by nature, nocturnal. Perhaps these are deer with better night vision that adapt better to feeding at night. Just a theory, mind you, but it seems to make sense.