I've always shot with fingers too and had the same dilemma. Two solutions I use are: 1.) Stuff a child's stocking-type headcovering in your shooting side pocket and keep your hand in there when not in use, or 2.) get a good pair of insulated gloves and just remove your hand from one before you need to shoot.

I have also seen some mittens that you can buy that have a "flip-top" over the finger portion. You can quickly flip the finger part open and it stays out of the way with velcro. I can't remember who sells them, but it should be easy to make one out of a normal mitten.