If it works, it works, but...

You're limiting yourself. Maybe not directly for your hunting situation, but a 12 would let you take longer shots and the slug is a little better at staying the course if it comes in contact with brush.
Reasons for my statement are; More gun powder in the 12, you could use a lighter slug and increase your distance. Brush does effect all projectiles, but, statistics have shown that the heavier the projectile and the blunter the tip, the better off you are if you were to come in contact with brush either on purpose or inadvertently.

I've never shot a deer with a 20 so I can't form an opinion either way. I do know that the 12 slug will fall below the 1200 (minimum standard for many) foot pounds of energy at around 75 to 100 yards and I would have to guess that the 20 would be significantly lower when compared to a given distance. Speed x weight = energy.

I can say this - I've never had a deer go more than 30 yards after being hit with a 1 oz Winchester Super X out of a 12.

Good luck,
KP