RE: Is a 20ga worth it for deer?
The questions should be, "Can buckshot from a 20 gauge (#3, I believe) kill deer?" Yes, they can. Then, the next question should be, "Will they give me the best chance of killing a deer?" Probably not. The hole made by a 20 gauge slug, even a sabot slug, will be more than enough to drop a deer, if you do your part and get the slug in the right spot.
Now, compare the energy that a couple of pellets of #3 buck. Not quite the same, and as Ballistictip alluded to, there's probably a few deer running around the woods with at least one buckshot pellet in them, because it didn't have the energy to kill cleanly. I'd say that this is a major reason why buckshot is illegal for deer hunting here in NY.