Sounds like you are a guy who plays by the rules of respect that most hunters observe... but this guy doesn't. I say there is only one way to handle an a$$ like this... confront him.
If he gives you anything but an appology... kick the hell out of him.
Now don't get me wrong... I am not John Wayne.... but hunters NEED to respect other hunters.
He not only is failing to give you any respect, he going WAY out of the way to disrespect you big time.
A sore nose will end that problem.
Good luck