Mottz is right. I had deer look directly at me from a close as 15 feet and not be able to determine I was a threat. You may have looked out of place but not enough to alarm him. You may have blown it but again you may not have. I would give the area a rest. If there are some doe around there he may come cruising the area later.
An arrow flying by wont necessarily spook him as long as it was a complete miss. You may have startled him. But if he never actually detected you - you could still be in good shape. Next time dont take an iffy shot. Its your responsibility as an ethical hunter to only take shot you are confident in. And even then there are things that can go wrong. An unseen twig, the animal ducks or lunges forward, all resulting in a less than perfect shot.