I know the feeling. A couple seasons ago, I had a big buck cross the field I hunt and into the cow field that borders the property. I had it in my scope broadside, wide open at 80 yards and didn't shoot. I don't know that the farmer would mind, but I had never asked him so I didn't pull the trigger.
Last season I saw the biggest buck I have ever seen while hunting. He was as big as about anything I have seen posted here. He followed some does out at last light within 20 yards of my blind. I was sitting there with him in the scope of my Knight Revolution. But, in the county I hunted, you have to shoot a doe before shooting a 2nd buck and I had shot a buck but no doe. I was absolutely sick. I have never gotten a wall hanger and this guy was a once in a lifetime buck, at least for me.
But I, and you, did the right thing and we have the memory and story to tell.