Here's the deal IMO. I agree that the difference in impact at 100 yds was due to cleaning or fouling.

The important shot in muzzleloading is the first. You should get a load you are happy with, and then for your fine tuning, simulate that "first shot" condition by having your gun the exact same as it will be for that first shot at game. Whether that's a clean dry bore, a fouled bore, whatever, it's up to you. Maybe that means you have to swab your bore after every shot, then dry it, then foul it. It's a pain, but if you want to know where that first bullet is going it's worth it.

It seems like some guns, especially inlines and sabots, are finicky. The first shot may be low, then 3-4 subsequent shots are "right on" then the next couple wander a bit and are really tough to load.

My gun does not shoot differently with a clean bore vs. a fouled bore, until shot #7 or so, so I hunt with a clean dry bore. I find that a hangfire can occur if your gun is loaded and the bore is fouled for a couple days. But that's with a hammer gun and regular #11 caps.