I say it depends on the type of rifle and slugs you are using. If it's a modern rifled barrel slug gun shooting newer slugs then I would actually suggest a 20 ga over a 12 ga. There just isn't enough difference between the two ballistically to take the punishment of a 12 ga. And from what I have seen the 20's tend to be more forgiving of different loads in accuracy.
Now if you are talking about shooting simple rifle slugs from a smoothbore gun then the 12 does have an advantage over the 20 as long as they are both fairly accurate. But if you can't shoot the bigger gun well then it really doesn't matter does it
. The 20 will kill deer dead if you put the lead in the right spot, it doesn't get much more simple than that.
Good luck and have fun,
Paul