It seems you're stuck on the H&R HandyRifle. It's not a bad gun, but there are better options for hunting. The main issue is that they are a single shot gun. I can't imagine hunting with one by choice, as any repeater will get you a second shot faster than a single shot. No hunter on earth can say they never have or never will want a fast follow up shot. If the cost of the gun is the real issue, maybe a deal could be found on a used repeater.
But if you are focused on the Handy Rifle, I'd go with the 44 Mag. It is ideal for deer in the woods with shots less than 100. I recommend the Hornady LeverEvolution round.
Shotgun slugs have gotten much better in recent years, but for your situation they would not be any better than the 44, and the 44 is much cheaper to shoot.