To start off, there are people out there who have killed deer, regularly, with 35-40# bows and a well placed shot. That said, whatever you're comfortable with works.
The heavier bow might come in handy if one were to get the chance at that special elk hunt or something like that.
Ten years, two shoulder and wrist surgeries, and about 80 lbs. ago, I had no trouble shooting 90+# compound bows and 60-70# recurve and long bows. This year, I'm just beginning to be able to shoot my 53# recurve well enough to hunt with and have my old Pro-Line Match Point target bow camoed and turned down to 47# so I can get in the woods. You never know what the future has in store for you.