ORIGINAL: SJAdventures
I am not sure what type of bench rest your using but I just read something the other day about having the actual baerrel in contact with what you are using for a rest. It said that if your barrel is resting on whatever your using for a rest instead of just the forend that it will throw flyers all over the place. Something about it messing up the harmonic vibrations or something to that effect. By what this stated make sure that the barrel is not resting directly on whatever your shooting it off of. Like most have already stated though usually it is something that is loose.
This is very true, holding the barrel or resting it on something dampens the barrel. Obviously, a tiny change in the way the barrel vibrates can have inches (or more) effect downrange. Another way to think of it is that the gun's points of contact when sighting in should be the same as when you're shooting in the field.