I'm not surprised at all that you're hittling on at 50 but 4-5 inches to the right at 100 yds with iron sights.

I would adjust the windage by moving the rear sight slightly to the left. Brace the gun firmly in a vise with some padding. Apply a strip of tape to the barrel, and another on the sight, close enough to each other so you can draw a line along the barrel across both pieces. That way, when you drift the base in the dovetail, you can tell how much you have move the sight. A minor adjustment is all that's needed.