Barrels get hot, barrels get fowled, and shooter get fatigued. There are a lot of things that might cause this to happen. I suggest when you start getting groups that begin to spread out, take a break. Really swab the rifle as clean as possible. Walk around. Think over what you were doing right, and try and pick out something that could have went wrong. That also gives the rifle a chance to cool down....