First off, check you State's game laws. Many have caliber restrictions for small game.
That aside, I would go with .45 caliber if it's allowed. You're going to be trying for 20 yard head shots anyway, and a down-loaded .45 is not going to destroy any more meat than a .32 or .36. With a hotter load the .45 is a better caliber for ground hogs. It will reach out farther than the small bores, and can be used for deer at an appropriate range. There's a broader selection of guns available, and components are more readily avaliable.