bflem55,
Are you sure you're still centered in your peep when the shot goes off. If you don't stay with the shot until it's gone and your eye isn't still centered, this can give you left and rights. Also, do you lean in to see through the peep? Keep your head level and bring the peep to your eye not your eye to the peep.