RE: Which caliber fits the bill???
The .243 was designed for exactly what you are asking for. A one gun compromise for a varmint gun and a deer gun.
I wouldn't buy one strictly for varmints or for deer hunting. But for one gun for both it does the trick.
I would like maybe a 6.5 X 55 or a .260 or a .257 Roberts or a 250/3000 savage better.
I do hear great things about the 7mm/08.
As I've said before, I usually fall in love with a rifle and buy it for that reason and not because of any logical reason.
Robin