If you're shooting Pyrodex, I'm betting that you'll get better accuracy if you swab between shots. Just because you're able to load the next bullet without swabbing doesn't necessarily mean that the fouling isn't causing accuracy issues. To be perfectly honest, for a scoped ML fired from a bench at 100 yards, those groups aren't that great. I'd be just barely satisfied with those groups at 200 yards from my Savage. Try allowing the barrel 2-3 minutes between shots to cool, and swab between shots for consistent bore condition, and you might see those groups shrink a bit.

Good luck,

Mike