I use the same sight as you do and I agree with bow scout you need to keep that shaft lubed. And you need to loosen the set screw up, but not too much to were there is any play in the shaft moving back and forth. And if you ever have to tighten the set screw don't tighten it too much the shaft then won't allow the sight to swing like in your case and you take a chance on striping the threads.
Good luck!