Lots of possiblities. Check all action screws and scope mount/ring screws. Torque action screws to 65 inch pounds. Do you know that there is nothing loose inside your scope? If you have a wood stock and/or if it isn't free floated, sometimes the heat of the barrel causes the stock to warp up or down (this would be your scenario), and cause the point of impact to shift as well. And if you started out with a real clean barrel, some barrels will wander up or down for the first few shots until they get properly fouled. This is particulary possible with non-premium barrels that may be a little rough inside.

What's the history of this gun? Has it shown tendencies in the past to wander itszero? If notwhile shooting a group, then printing a group in a different spot as the weather and temperature changed?