Yearlings are not too terribly smart at times. Yes it sounds like they got nervous, but they probably weren't spooked. I'd hunt on. Like you say, you've got limited time. Unless you have another stand that he may pass by I'd go for it. You certainly won't kill him sitting in the house in the next 3 days. If you're in a tree.... you've got a chance. They got a choice food source and will hit it. The rut is at hand and the bucks will be trolling. If you got does around the bucks will be close. Go for it.