Many people do as you do and never have a problem. However, Giardia and other things are always possible, even in the high country. Just rember, unless you get your drinking water directy from a spring, there are going to be other thing in it, like a deer/elk that pooped up stream from you. How about elk/deer,marmot, miceurine?

A water filter is pretty cheap insurance. I bought a MSR miniworks and sleep better at night.