Myself, I went from 68# compound with 50% letoff to 50# in a cheap and used but good recurve and it was all I wanted to handle at the time. If I were in your possition, I'd get an used but solid bow around 40-45# at your draw.

Your draw may change going from the compound to a recurve or longbow so you may want to get your draw length checked on a low weight stick before doing anything.

With a lower weight bow, you'll maybe want to use a heavier grain per # of draw weight arrow and scary sharp 2 blade broadhead. It will maximize the energy of the bow that's transfered to the arrow and the broadhead will minimize the amount of contact taking place as it cuts it's way through the animal, helping you to get the best penetration you can from the combo that you're using.

Most importantly, have fun. Cause that's what it's all about.