Before you discuss ammo and trigger job and all of the misc info, you should post what are you are using for a rest. You stated you are using the best way to steady your rifle you can find. If its just a rolled up shirt or gun case then 2" groups are more then fine. Without a very good solid rest its a question of how well you can shoot more then the gun and ammo.
Until I dropped the gun into a 100% solid rest and tested it out I wouldn't worry about the sub moa accuracy of the rifle and would very confidentally walk into the woods with a 2" group from just leaning the gun on something and shooting it. Obviously the gun is shooting fine at that point. I mean your deer hunting not shooting 10spots.
If I can shoot 5" groups while standing up shooting offhanded with Rem ammo I doubt the ammo would be the first thing I worry about tightening up my groups.