That's definately not a good shot to take(straight down). If you give him time and a shot presents itself(such as a quatering away) then take it then but shooting straight down on a deer is a bad decision. If the shot never gets better and the deer disappears then that's hunting and the deer wins a round.

You made a good decision by letting the deer pass. The shot wasn't there and you didn't force it. Kudos