Depending on the powder you used... water could clean it, or solvent. Take some solvent on a patch and with a good tight fitting jag, scrub that area. If that fails get some JB Bore Paste and use that to clean the rifle. Next time, don't wait a few days to clean your rifle. Some powders are fussy about that.

Also swabbing between shots is a good thing, but again... depends on the powder you were using.

What kind of cleaner were you using?