What you are experiencing is not all that uncommon. I have rifles that shoot different on a clean barrel and then again on a fouled barrel. What I do is decide what I want to do. Since I shoot a lot, I sight them in on a fouled barrel. Before I hunt, I just shoot a squib load. Dump 30 grains of powder down the barrel, push a cotton patch on top of that and fire it off. Now the barrel is fouled and should shoot like it normally does.
Others hunt on a clean barrel and sight in for a clean barrel. Although they do note what the fouled shot hits. Another thing you might try is, if you shoot on a clean barrel. When you get done shooting the first shot, do an extra extra good job of swabbing the barrel. Almost to the point of it being clean. Then shoot it.