You should never say never with deer, but, deer shot with rifles never run 40 yds and stop to look back to see what bothered them.

That said, always take the time to really follow up a shot to determine what happened. I've seen more than a couple deer recovered after a "miss".

My guess is that the bullet hit the ground near the deer and he spooked.