I've packed in just like you...climber on back with backpack on front backwards. Then I said screw this and culled a few pounds of gear that I really didn't need.

Now I just use a fanny pack along with my climber on my back.