If you were shooting your shots in rapid order the barrel would have heated up and expanded such that there may be a spot in the barrel channel that is applying a different pressure to the barrel when it heats up , thus changing the POI. Take your time between shots. If you start to notice that it is harder to load the next ball, you may need to swab the bore after so many shot to maintain accuracy. Several of my guns require that I swab the bore after 2-3 shots. If I don't, they are much harder to load, and the POI is different from the first 2-3 shots.