If what you are using is successful, stick with it. There is nothing wrong with the 20 gauge for deer.
That being said, I never notice the recoil when hunting. Admittedly, I am not terribly recoil sensitive but sometimes I cannot remember the noise. I am too focused on what I am doing.