Give the bow a good looking over. Make SURE everything is tight, and eliminate the possibility of something being loose and moved. Check for cam lean and limb twist, but I don't think you'll find either one of those.

I highly suspect bow torque. Does your sight have a level and do you use it? Torquing the bow will cause left/right impact.