I do not use a harness. Years of climbing up and down ladders while working on Construction jobs has taught me a thing or two about safety without the use of a harness. Accidents can happen to anyone, even those who wear a harness.
The last thing you need is for a harness to SOMEHOW get tangled around your neck as you fall. If it happens, you're most likely dead.
We all have our personal choices, and I can guarantee you that a lot of us do not wear a harness. It does not matter how "safe" you think you are, accidents happen to the best of us.