I have wondered about this as well. If you shoot a deer rifle at the range with no ear protection you will regret it. You shoot at a deer and you don't even notice it. So is it doing damage and you just don't realize it?
I do know that a muzzlebrake will split your ears when shooting at deer. My Browning BAR came with the BOSS system which uses a muzzlebrake. The first time I shot a deer with it it rang my ears good. I replaced the muzzlebrake with a solid piece and generally don't have that problem now. I do notice the shot more when shooting from a blind. I guess the blind keeps some of the noise in.