Don't marry your bait pile.

While the deer detour to visit it, they sound as if they're on the way someplace else. If they're only visiting your bait at night that's a good indication that it's in area they are not comfortable with.

You need to figure out where they bed, and ideally, find the point(s)and trails where they divert away from your setup. That's where I'd wait.

Good Luck