Something is loose somewhere.You shouldn't have the pin off center very much at all.I know you don't want to hear this,but take it completely apart,and assemble it with lock tite BLUE.It will stop the parts from vibrating loose.If it still rattles loose then something is not clamping down properly,and you may need to replace that part.(Sight,Rest)