I also clean between shots, but I always fire off three primers and dry swab the barrel prior to loading. This slightly fouls the barrel and burns off any remaining oils, solvents, etc... This is the routine I use when pulling the gun out of the safe, so I do not mind the time it takes at the range. It also stopped any first shot fliers.

When I was using T7 pellets in my CVA Kodiak and 245 gr PowerBelts, I had a terrible crud ring. Swabbing the barrel did not completely eliminate the ring and caused inconsistencies as I could not seat the bullet with to the same depth and pressure each time. Just a thought, but make sure you are gettin the bullet seated the same each time. I had to use a brass cleaning jag prior to switching to BH 209.

You have stumbled on the most frustrating and enjoyable part of muzzleloading. Load work-up. 250 Shockwave / SST's or XTP's are known for their accuracy, readily available and inexpensive. May want to start their with 90 gr and work the powder up by 5 increments each 3 - 5 shot group. You can then move towards some of the premium bullets mentioned after you have found a reliable starting point & fallback option.