Often deer don't know where the **** the sound came from! Ever watch other deer after a shot? Some just stand there, like "What's going on and why is that buck running away?" LOL!
I've seen plenty of deer right at a firing range! Deer get acclimated to just about anything... humans, cars, sounds, smells, sights, etc. Your neighbor is a little naive and you should either show him this post(LOL!) or try and politely share with him how deer react.
He does have a point though. Deer, bucks especially, will have some uneasiness to an area if shooting continues for a while where there was no shooting before. It's sort of like in between what I'm saying. A lot of shooting, and we're talking over time, the deer will get use to it. If we're talking a few shots over a day or so from the same place, the deer will have to get use to it, but will take some time.
Deer will be less tolerant to your hunting/walking sounds and the scent you leave behind than a few shots. I don't think deer adjust their patterns, per se', to gun shots, but to the overall intrusion into their area.
iSnipe