If you can hit a groundhog at 150yds with a 22-250 consistently and you are willing to pass on marginal shots, then using a small caliber centerfire is fine.

The problem is that most lead-flingers don't wait to pick their shot, or they hunt where deer are constantly moving.

If you hunt where 1/2 of your shots are at moving deer, then usinga small caliber is pointless.