The thing about muzzle brakes, much like T/C QLA, is if its not centered it can cause accuracy issues.
You hear pros and cons about barrel break-in. And in your case, this is a strong case for it and my reasoning for it. My thoughts is that a break in is necessary for better accuracy. Not right out of the box but down the road. Had a break in been done initially, the minor burrs and scratches in that barrel would have been removed allowing for easier cleaning with less copper build up after shooting. As you proved it was this build up plus the muzzle brake that caused the accuracy issues.
You may want to have a smith either remove the threaded area of the muzzle and re-crown it or install a nut over the threads to give it a finished look. Glad you got it shooting again.