Using Wonderlube it can happen. What you will find is if the patch is frozen in the bore the accuracy can suffer a little and the patch will usually be blown to bits. There is really no way to guarantee that it will not happen when you hunt in sub zero temperatures.
I've hunted Wisconsin for more years then I care to mention. We've been out chasing whitetailin -30 below and we are not talking wind chill. Yes it happens. Do I worry about it? Not really, I make my first shot count. You might find things a little stiff when loading the second shot but it can be done. Keep your ball and patch in a small tin and keep that near your body. The patches will not be very stiff orhard as a rock then if you should need to load a second one.
I've since switched patch lube to the castor oil, alcohol, water, murphy's oil soap, and witch hazel mix known as Moose Milk. I think the alcohol will help keep the patch a little more cold weather friendly. So far I've been lucky and have not had to test my theory because now when it gets too cold, I just stay home...