Like Mark said, it could be a burr in the barrel. If I were shooting those kind of groups at 100 yards I would not care what the sabot looked like.

If the sabot thing is really bothering you, try some other sabots and see if the same thing happens. If it does, it is probably a burr. Get some J-B Bore Paste and scrub the barrel of your rifle real good if you're that concerned. If it is a burr that Bore Paste will smooth that right out while at the same time take the lead, copper and plastic fowling out of the barrel. Be sure and oil the barrel good afterward.