Nope.. you're fine. What your experiencing is that when the barrel is real clean and dry you get that plastic trying to engage the rifling on the way down. Next time you load on a swabbed barrel wet the outside of the sabot with spit. Just roll the sabot outside in your mouth and get it wet. That little amount of water will be gone before it gets to the powder and will help it load a little easier.

After you have fowled the barrel there is a buildup of fowling in there that if you do not swab acts as almost a lube of sorts. The problem is if you let that build up it is more of a mess to get out, and a it closes in the barrel will make loading harder. I often times swab every third shot depending on the sabot and bullet. The SSB's by Buffalo are one that I can load about three before I have to swab.