For a " perfect" shot within 100 yards, you could probably get away with using a .22 RF. Shots are seldom " perfect" though, so I would not recommend anything less than a .243, and thats with good 100 grain hunting bullets! Anything less is just not fair to a fine game animal like a Whitetail deer!
In respect to the game we hunt, we owe them enough respect to use a weapon that will make a clean, humane kill, 100% of the time! While I believe that you can never be " over-gunned" , you can be " under-gunned!"