Have you thought about a lever-action 30-30? Great little guns, not much recoil, inexpensive (especially used ones), perfectly suitable on deer out to 100 or so yards.

As for recoil, I'm a big guy too, and I can take a .30-06 with no trouble--just remember to really tuck that rifle into your shoulder when you shoot. If you don't, it'll accelerate into your flesh, which is the kind of thing that'll spoil your shooting day in a hurry.