You never "have" to tumble brass. You should clean it up a little before you run it thru your dies to keep from scratching up the inside of them.
The first 25 years I handloaded I didn't tumble brass and even today I only do certain rounds. Getting the tumbling media out of .20 cal brass or anything with a sharp shoulder is a big PITA.