The 7mm Mag should drop a deer with no problem with the right bullet. I would guess you are using a bullet that doesnt expand enought. You can get away with poor bullet choice a lot more with the larger caliber bullets due to the inherently larger wound path created(assuming the same bullet construction). At any rate I choose the 06 also as my hunting rifle and have zero regrets.
I recently took a client on a Russian Boar hunt and he took a 400 lbs hog and had results similarto yours with a 7mm mag, 160 Grain NP. The bulletspassed thru with minimaldamage. Tons of penetration butlittle energy transferred to the animal.