I have always used #4 for the down range energy. I figure if its close I have plenty of pattern, if its further out there a good pattern won't help much if the lighter load has lost all its juice on the trip.
I took my last bird with a low-brass crappy duck load cause thats all the store in town had, but I was aware and knew that I wouldn't take any long shots.
Now I'm carrying a hybrid 4/6 load but I haven't had a chance to pattern them yet, but its gotta be better than what I was using.