No spot isnt ruined if you left it be for a few days. By the reaction the deer showed, it would appear he may have scented you but he didnt fully get scared. He caught scent and jump but stopped. That tells me he is not entirely sure if the scent is still present or where a threat could be so he just did what they do best and returned to thicker hiding.

Now that goes to finding out what you did next could play a better role in deciding. Did you sit for the remainder of the hunt or did you just call it a day and immediately get up and leave.