My guess is the deer came in to see what was grunting, saw no buck and turned to leave. I've set up with very little cover at the base of my tree or on a field line and had deer hang up to 50 yards or greater. They look and and just won't make the commitment to come in another 30 yards. Either only set up where you have an obstruction that the deer you are trying to pretend you are is hiding behind (does this make sense?) or use a decoy. Other than that if your close to the rut and the deer is big enough try a snort weeze, this sometimes will piss them off enough to come in anyways. I like to think of myself as the deer you are hunting... what i mean is, if I heard someone yell hey in the woods -- but I couldn't see anyone where it came from, I would think this very strange and probably walk away slowly / looking back and confused. But if they yelled get the heck out of here, angrly (snort weeze) I might feel like seeing what was going on a little closer.