What you will be gaining is a few hundred fps and another rifle. This extra speed will result in more recoil. Your '06 is fine for mulies and elk out to 300 yds and beyond. If you want to go bigger you might as well jump up to the .338 Win Mag.
I recall an old adage: "Beware of the man who only owns one rifle - he probably knows how to use it."