This will happen alot if they come in and dont see another deer they usually wont come closer. A decoy might be something for you to look into. Or you said when you grunted he lost interest, did you try to keep grunting while he was coming?