If the deer think they have a refuge or you have found their escape route form other hunters I would go back. Always go back to the place you have seen deer rather then a place you haven't. Just watch your scent and don't spook them with anything other then a gun blast.