The 243 will work for both, but IMHO there are better calibers to choose from. I would buy two rifles. The 223 would be my pick for varmits. The 270-winchester would be my pick for a deer rifle.
The 25-06 would also make a fine varmit/deer rifle.
If you ever plan on hunting anything larger than deer? ie: Elk, bears, moose? You might want to add a third rifle or buy a larger caliber second rifle? Rifles chambered in .338 caliber & up are my pick for the larger critters.
You really could do it all with a 338-win mag, 338-R.U.M. or 340-Weatherby.