Sound to me like there is not enough down pressure on the rear sight itself. I once fixed a similar problem I had like yours by:
Remove the sight elevator
Lift up the back of the rear sight, and slide a wooden matchstick or something similar as far forward as you can, elevating the rear sight a good bit. Then, with thumb or finger pressure, press downward until the sight actually bends slightly. Remove fulcrum material, re-install elevator. It should be much tighter now.
Hope this can work for your particular sight and situation.