how often do you guys use the wire brushes down your barrells after shooting?

I had done it for the first time last weekend, and it was crazy because its starting to take longer and longer every time to get all of the "black" out of the barrel. I don't shoot much at all, and have only had the rifle for a few years.

But i had used the foaming bore cleaner, then put patches of rem oil through, and just kept rotating between wet and dry patches and it finally was a very light light grey, then i scrubbed it, and it was black again!.

But im worried about ruining the barrel by scrubbing too much.

After leaving it for about 3 years without a good scrubbing, could this have caused any damage to my rifle? It seems to be shooting fine.