I haven't shot any really large rifles, 30-06 is about the biggest I think in a centerfire. And a lot of it really depends on the rifle. I had a savage 110E in 30-06 when I was younger and it didn't bother me at all. And I was only like 5'3" and weighed about 85-90 lbs at the time. My step dad had a Remington pump in 30-06 as well and that thing was BRUTAL to shoot. I hated shooting it.
I have shot a lot of magnum muzzle loaders and larger shotguns with no real problems though. Like H&R ultras with three inch magnum loads and 600 grn slugs, 870's and 500's with winchester platinums and the likes and various turkey guns like 3.5 inch pumps with full turkey loads.
I even shot a 3 inch high velocity turkey shell with 2 OZ of #4 shot out of a standard H&R tracker single shot. That is one I probably won't forget soon

[:@].
My thresh hold is probably 3 inch turkey loads out of a standard pump gun like a 870 or 500. I can shoot a 3.5 inch gun, but it's not something I care to do often.
I bet I could probably shoot a 300 win mag without many problems. I wouldn't be afraid to try it I know that.
Paul