I also layer, long johns then lots of camo.
Hunting the Rocky's you can start out in the 20's and then it heats up to 60 or more and cooling down again after 3 pm. I always have either extra space in my pack to store shirts or I can hang shirts and a jacket over my pack then put them back on later in the day.