I'm not sure i don't agree with the premise of hunting so close to the property line of a known neighbour who would refuse access in the event of a wounded animal??? I'm not saying I agree with what he said or how he said it but the underlying thought isn't that crazy. I have had the exact same situation where the neighbours made no bones if I stepped on his property they'd have me charged. In all cases I have chose other locations that wouldn't put me into such a situation. Why play with fire?? Here the letter of the law is if you can prove the animal was hit prior to crossing the property line you can retrieve it with a CO/DNR ******. No permission or CO ****** is trespassing, even if you can show it was taken legally (the landowner has the power to say charge or no charge!).
You did the right thing you asked permission to follow up on a wounded deer. Prior transgressions don't give you the right to even the score. You also did right by not just going anyway. The situation itself sucks and feel bad for you, your son and the lost deer. Hopefully you can come to some sort of agreement or understanding for the future.
Best of Luck