It's also worth pointing out that there is a big difference between a 250 pound northern bruiser of a buck, and a 90 pound southern doe. I'd feel pretty comfortable with a .223 with a 60 grain Nosler Partition or comparable bullet for smallish to medium-sized deer, but if I was going in an area that was known for very large deer, I'd probably want something bigger.
Having said that, if I did have a big buck in my sights and a .223 in my hand, I'd aim for a neck shot and it is lights out. I've used neck shots from time to time and given the proper situation (unmoving deer, close range, accurate rifle, good rest) they are absolutely deadly.