I see some positives in the situation. That back was confused plain and simple. It dosn't sound like he knew what you were and it dosn't sound as though he was really spooked because he hung around so I don't think you blew the spot. I was busted earlier this year becasue I decided to get down 5 minutes early. It happens and we learn from it.

If I were you I'd be back in that tree the next WNW wind you get. This time just stay until dark