It sounds like your swabbing technique is just fine. I think the main reason to swab is to get rid of powder fouling to enable easier loading for the next shot and what you are doing should accomplish that.