ORIGINAL: Pawildman
Sound to me like there is not enough down pressure on the rear sight itself. I once fixed a similar problem I had like yours by:
Remove the sight elevator
Lift up the back of the rear sight, and slide a wooden matchstick or something similar as far forward as you can, elevating the rear sight a good bit. Then, with thumb or finger pressure, press downward until the sight actually bends slightly. Remove fulcrum material, re-install elevator. It should be much tighter now.
Hope this can work for your particular sight and situation.
I took a closer look at it today and I think the problem is with the ramp itself. There is lots of down pressure on the sight but the notches in the ramp seem shallow. I'll try to bend the sight down and see what happens.