Not sure why your Handi rifle is shooting so poorly. These rifles usually are fairly accurate. Not being there I could offer some suggestions. Like some of the posts said use sandbags instead of the gun vice. If you don't have access to any sandbags, get a couple bags of cheap birdseed and fill up some old tube socks. They will work and are much lighter than sandbags. Secondly, be sure to only rest the gun on the forearm and not the barrel. And try to be consistant from shot to shot with cheek placement and pressure.
And you said the rifle is knocking the spit out of you! It is possible that you are anticipating the shot and subconsciously flinching and not knowing it. Try placing a pair of gloves or some other padding under your shirt to soak up some of the recoil. The '06 really isn't too bad of a kicker.