This is only my second year at this but here's what I do...

If the temps are colder, I'll wear my harness underneath my outermost layer. That way, in the off chance I get too warm, shedding a layer isn't a big deal. Plus, it would make the blaze orange thing easy to deal with.