If it's a Big Boy you're after, the hunt him! Not uncommon for the more mature deer to hang back a little deeper. The area sounds like a good one. The key for an afternoon hunt would be to sneak in undetected without disturbing the bedding area. If the wind is right and this can be done, go for it. The thing about morning hunts is that you really have to get in the stand early, especially when the bedding area is not far from a stand location, like yours in this case. I have walked in on some deer heading to the bedding area and jumped them before. If I had been a little bit earlier, they probably wouldn't have spooked like they did.