Generally, in most patterning situations, the smaller the shot size - the tighter the pattern. If you purchase 4's to tighten your pattern, you will not succeed in doing so. Always keep in mind when you reduce the pellet size, you also generally reduce the effective killing range. My rule of thumb is, if your gun will pattern 4's well enough to get a total of at least 12 pellets in the head and neck kill zone at 40 yards, then you have a turkey killing gun. You will likely see an increase in pellet count inside the kill zone with a tighter choke and using 6's, but the question is, do you want more pellets or more killing power. 5's are a good happy medium between good pellet count and good killing power. Sounds like you gun shoots 5's very well at 20yds. I had the same type of problem with my 870 last year. I improved the pellet count in the kill zone at 40yds by simply choosing to shoot high velocity 5's instead of standard magnum. The problem I was having was that I was trying to push to many pellets through the smaller choke. High velocity shells have slightly less pellets per shell than standard magnum shells, but have more powder for faster and harder hitting charges. This makes for a tighter more uniform pattern.

Let us know how it goes.