Another .243 vote. I've shot deer with both, either will work fine. But the .243 gives you allot more range, and more use in my opinion. To me the .243 is a deer rifle. And it sounds like you need a dedicated deer rifle. The .44 mag is a backup type rifle, or fun rifle. I grab a .44 if I'm antlerless hunting or if the weather is crappy and I'm driving for others. I have a handful of open sight lever actions that I use for that. But if I'm serious about getting a deer I'd grab the .243 every time.
-Jake

Edit, that's a .44 in my picture there, kills squirrels and deer both just fine. But I wouldn't want it as my primary deer rifle.