Sound like from your description you are moving your scope setting fairly often.
If you leave your scope setting alone for 4 or 5 trips to the range do your groups settle down.
When you adjust your scope make sure you check it after adjusting it to make sure the zero actually moved to where you think you moved it,
If you are adjusting your scope after you finish a shooting session, you will have to recheck and adjust your zero each time you start your next shooting session.
Sometimes when you adjust your scope setting it may take a shot or two to settle in, especially if it is cold. Also returning your scope to the previous setting may not actually move your point of impact back to the exact previous spot.
Try leaving the scope settings alone for awhile and see if your points of impact don' t settle down.
If they do settle down your scope is ok, if not then go on to loose mounts on your scope, you cannot tighten your scope down to tight either, shoot from a dirty barrel for a few sessions, cleaning residue in your barrel can affect impact, check the bolts holding action to stock too tight or too loose can have an impact.
Are you shooting the same ammo each time, change anything in your reloads or factory ammo and impact will shift.