On deer I wouldn't use anything smaller than a .22 caliber center fire. Main reason is that they don't make (as far as i know) proper bullets in sub .22 caliber cartridges.
Ive heard so many stories of faliures over the years, from various calibers, then find out later in the conversation, the hunter was using something like a 180gr A-frame style bullet in a 300mag.
It probably just zipped right through the 90lb deer without veering or expanding, making a little hole. A 150gr cheapo Core-lokt would've dropped the deer.
I have used my .338RUM with 250 grain A-frames to take a couple of deer here in Florida. Massive trauma inside the deer with very minimal meat damage.