Any of the calibers you mentioned will get the job done for what you're hunting. Especially at those ranges.

Longer ranges might dictate the upgrade, but if you're looking for an excuse to buy a larger caliber, just make something up and do it. Can never go wrong buying another rifle if you can afford it! Both the .223 and .204 have proven themselves.

Only thing I would say between the two is that .223 might be easier to get ahold of. But the .204 is getting really popular. Don't have one, so I didn't notice, but I suspect they're not too hard to find either.