Resting the barrel directly on the sandbag will invariably cause poor grouping. You should always rest the forearm on the bag, and always in the same spot to try to get as much repeatability as possible.
As far as the forearm seeming to be loose or sloppy, are you certain the action screws are tight? If the barrel and action are flopping around in the stock when fired, accuracy will be terrible. You might take a close look at this..........