That's painful, but you gotta do what you feel comfortable with and not let other people get you down. I had a slightly similar situation on Sunday. I was hunting here in WI and had a lone doe come in just before dark, ~7:05 as best I can figure. I moved slightly to try and get her on film and she pegged me right away. She backed out a little bit gave me another shot opportunity at 20 yards broadside in the exact same location I shot a doe the weekend before. I was at full draw on her, but she pegged me again and I didn't feel comfortable letting the arrow fly in the low light on a deer that was already spooked. It got down to the low 30s that night and I'm sure she would've been fine overnight if predators didn't find her first, but with a full day of work ahead of me on Monday, I decided I wasn't comfortable with it and didn't shoot. I regret it a little bit, but looking back I feel confident I made the right decision.
Not trying to put you down or anything, I know the pain of not recovering a deer, or recovering one that's spoiled. Just sharing some anecdotal information. Good luck on future hunts doing what you enjoy.