I had this happen to me as well... I was going to actually take the momma doe to a button buck, and he looked up at me when I was going to take her, even after standing in front of me for 25 minutes before she showed up. The next day and the day after that was the opposite, she busted me and he never even looked up. It happens to the best of us, you just cant out smart some deer.