The 180 grain bullet is a prettygoodbullet for the 30-06. While the 168 grain is the flatest shooting, the 180 gives you the best of both worlds...superior bullet penetration and trajectory. 150s will work fine on deer, but the inch or two of trajectory advantage does not overcome the 180's ability to penetrate. Having considered the 30-06 to be "my go anywhere and shoot anything rifle", I prefer to use a standard load so the I am well aware of my rifle's trajectory. I doubt seriously that your ammunition is the culprit.

Most likely, you either have a rifle problem, optics problem, or operator error. Being anal about good shooting rifles, I would check your action screws. Are they tight? I would check your scope mounts/rings. Are they tight? I would shoot multiple groups at paper and look for flyers or shifts in point of aim.