To answer your original question...Yes, ammo can make that much difference, even if everything else mentioned is good...
I have a .243 with a custom target barrel that just hates Hornady 100grs, Winchester Supreme 95grs and Federal Fusion 95grs...I'm talking 3-4 inch groups from a gun that will group other ammo into a dime at 100 yards...

Keep working at it and it will all come together for you...