first of all it really seems like you made a decent effort to explain to him what happened and that you didn't mean to be on his property. He was just beeing a dummie.
After following up with him and he was still like that I would definetly not allow him or his family on your property and I would more than likely at that point had enough of his loud mouth and gone over to take him up on thebutt kicking.....remember it is better to give than receive and he would be recieving if he didnt shut his pie hole....

people like that piss me off and I know I should just keep quit and let them be but somebody has to put them in their place or they will continue