I would make sure your peep sight isn't moving. If it were to slide down slightly, it would lower the point of impact.
Another good ideayou may already know about..is
once you've fine tuned your setup, take some white out and mark reference points on your sight and rest adjustments.
gl