You should use at least a .357 Mag 158 gr. softpoint. The .44 Mag is very popular for deer, but .41 Mag is a good in-between caliber for the recoil-sensitive.

Get a decent handgun scope if you can; it can really come in handy.