I'm not that old (33) and I keep my self in pretty good shape, but I don't find it fun at all to haul a deer out of the woods. I usually go back to my truck get my cooler on wheels with a handle and head on back. Butcher him right there and leave. Of course if you live somewhere where you need to take him to a check station that's another problem, but in VA you just call the animal in and tear a corner off of the tag.

P.S. Throw a bungie cord in the cooler too, that way if you want to keep the skull you don't have to carry it in your hand the whole way back. Just strap to the top of the cooler.