Vane contact...cam lean....bow torque....any of these are possible causes of your problem......also, the amount you should be moving your rest while shooting 20yds should only be in about 1/16th adjustments...

note: even with a perfectly tunned bow and steady arm and great arrows ... sometimes some broadheads just won't fly spot on with FP's (but should be closer than 3")