One variable you didn't mention, that would make a difference too me in this decision, is if this was a bigger/mature buck.
Meaning, over 3 1/2 years old and if this was the case I probably would pass on the Doe, as not to spook him.

However, that early in the season (for N.H. it's 9/15), I would venture to believe it was a smaller buck (maybe 1 1/2 years old)
and the answer would be yes, I would shoot the Doe.