In a 30-06 the 150-165 grain bullets are constructed for deer, the 180s for larger game like elk...This means the 180s might not open up as well on light weight game like deer...They are constructed to be tougher to penetrate further into a larger animal...
Use the 150s, they will do more damage and shoot a bit flatter...