All good points, I agree with each of them. In my defense, this hunter did actually move closer to me than where I originally ran into him. (not sure why)
Each scenario is certainly a case by case basis, especially since there's no regulation that I can find up here in VT. I do believe I would have had no trouble recovering the deer with my first shot, and if my situation and the other hunter's were reversed, I would've had no hesitation giving up the deer as the 2nd shooter.
Regardless, at the end of the day there was no argument and the deer was harvested quickly. It also built a good relationship for the next time I hunt at that location.