I do feel the rest makes a big difference but I think it is more in the way the rest is addressed. When you shoot off sand bags (at least I do) I find myself much flatter to the table then when I shoot of a Caldwell bag or rest, where I am sitting more in an upright position. So when you start changing the manner in your hold, it makes sense that the groups should change also.

I use a light weight Caldwell rest and simply let the rifle kick me and move me around. It is the best why I know how to make things back to the same. I have spots on the rest and the table where parts of my forearms and elbows all need to be in order to shoot from the same position each and every time.