Sounds to me like the deer patterned you. And they knew that once the sun went down, they had a nice free safe meal at your bait pile.

Next season, have a variety of stands. And if you bait, only bait for 2 or 3 days and then hunt. Use smaller amounts, like a gallon or two so the deer have a reason to compete at bit and arrive early rather than after dark. And try to bring your bait out when the deer are not likely to be there - if you are only baiting a few days, you can probably do so without them knowing. If not successful, move to another stand and do the same. Choose your stand and hunt so you are downwind from the deer. The key is the element of surprise.

And if I were you, I would shoot the first legal deer you have a good shot at - this will help you get some confidence.