It doesn't sound like you spooked him bad at all. Spots like this that i have i like to save for absolute ideal conditions. I don't take chances with these spots the wind must be absolutely perfect and the weather fronts must be working to my advantage. I would say wait at least a week and hunt it on the next available perfect day. I'm just throwing that out there to make sure that all your scent has dissapated, if you feel like it could be hunted before that by all means do!!! Post pics when you kill him!!!!