You need to shoot what you are going to be happy with. It did not sound like you would have been happy with that deer.
So in my opinion I think you did the right thing. Just keep at it and you will find the right deer. Good luck to you.