It's what gets embedded in the surface that's the problem....

When starting with a perfect surface, and actually doing a real hand rubbed finish...there's no imperfections to fret.