Sounds like you may have a problem with the bedding of your rifle. But first you could try a different type of ammo to see if it makes a difference. If you are using your rifle for hunting then most first hunting shots are shot from a clean cold barrel so thats how you want to know your bullet react. Fouling shots are used on the range for competion shooting but seldom are they used before hunting as this could require leaving your rifle dirty for days on end or firing a couple of shots each morning.

Best Wishes