Jeff,

If that spot is close to other human activity,(knowing where you hunt)I think you will be fine. He probably just got a wiff of somelingering scent. And it doesn't take much to make them nervous. I don't think he really figured you out or he wouldn't have stuck around. He can still be had there.