While I have never hunted with a shotgun for deer (I'm a muzzleloader guy, at least until I get out to a rifle state), but my dad hunted for many years with a shotgun before switching to muzzleloaders, and the gun he used exclusively (because it was the only one he had) was a Remington 870 Wingmaster 20 gauge with the fixed choke smooth bore barrel. He used Remington Sluggers because they were the only slugs he could afford, and he killed many dozen deer with that gun as dead as dead can be.

Look at it like this; and hit that will kill a deer with a 12 gauge will kill a deer with a 20, period. It's still a big chunk of lead making a big hole. As long as you don't push the ranges much past 100 yards, you'll be fine.

Mike