You know for a fact thathe beds there, and from what it sound like he wasn't sure what you were or else he would've been long gone, so he will probably continue bedding there.

I would set up a tree for your stand and then let it rest for while.It sounded like he was returning to his bedding area after a night of feeding, so I think that you were just in the woods too late. If you would have been in the woods an hour earlier you could've been sitting and ready. So I'd givethat spotbreak and just get in extra early to ensure that you don't spook him, and preping a tree will make getting up and in your stand much quieter, and make less chance of spooking a deer.