Thinking on this a bit, how long have you had this rifle? How many times have you shot a heavier caliber than say a 30.06?
I found after several range sessions that many of my fliers were because of me, not the rifle.
It has been shown to me, by the rifle, that it is far more accurate than I am, with it (this is usually the case with most hunters).
It might save you many hundreds of dollars to spend more time with it at the range, and get to know your weapon first, before you invest the money on glass and a new trigger.

What I do, when I go tot he range is take a medium towel with me, and fold it over a little to simulate the heavy clothing I wear while hunting (think northern MN and WI in November). Doing this greatly increased my accuracy, with all loads tested. I also invested in a quality range bag to help prevent slipping during the shot.