Thats the only problem I have with hunting with a large caliber pistol, the noise!
Of course you can wear earplugs, but then you're handicapped against the deer. I myself, have to be able to hear in the woods.
You either save your hearing and go without hearing the deer, or you just tough it out and blast your drums out.
Other than that, it is fun to handgun hunt. I don't wear earplugs when I do handgun hunt. Also, try a 300 grain Speer bullet if you handload. If they will group well in your pistol, they are deadly on deer. The extra weight helps. They are finicky with 300 grain bullets for the most part though but some will shoot them good.
Being able to be deadly with a pistol is about like bowhunting. It takes alot of practice and if you lay off awhile, it's like you have never shot a pistol when you start back. Just have to shoot alot. I recommend handloading if you don't already.