I disagree. Equal access, as you stated, is equal access. That means he has just as much right to be there as you, regardless of who gets there first. Im not trying to be a jerk, but I hear a lot of people say, "I was there first." That is simple not a valid argument. True, he could have dont the respectful thing and walked away, but, if he has access, you have to suck it up and deal with it like a man.
I shot one of my larger bucks about ten minutes after someone walked right under my treestand. Buck didnt seem to mind at all. Just goes to show you anything can happen.