My Tikka T3 Lite Stainless in 300 WSM knocked some sense into my little brother. He is 6' 2" and 180 pounds. He only shoots rifles that are a 270 or smaller now. im 5' 10" and 155 pounds and i have no problem with it. If you know how to shoot and handle recoil it wont be a problem but if you dont know how to properly handle the recoil you will regret pulling the trigger.
There was a guy at my gun club that is probably 6' and all of 300+ lbs and was shooting one of those and a 300 RUM custom. He said shooting the 300 rum was much better..it has quite a bit of muzzle jump.