For deer only I'd get a 7mm/08, but you said you might go for elk occaisionally. So I'd recommend a 30/06 or the 7mm mag.
OR...you could buy a used Savage 110 in 25/06, 270, or 30/06...and then buy the 338/06 or 35 Whelen barrel swap rig from Midway..........for about the cost of a one real nice new gun you could have the versatitlty of 2 great used guns for different purposes. You wouldn't need to change anything but the barrel for your swap....your bolt face and follower and magazine would all be the same. Just need the go, no-go gauges for , the 338/06 barrel (or 35 Whelen), the barrel wrench, and a barrel vice.
the only drawback is that you'd have to get the gun set-up with a couple trips to the range each and every time you wanted to swap it out from the little bore to the medium bore. one scope should do you just fine. Just leave it on the gun and make your adjustments after the barrel installation.