One thing to consider on the front sight height. You're only shooting at 25 yds, so its hard to say what is really going on, but...changing two things at once (rear sight andcharge) can be confusing.

I would expect a reduced charge to hit HIGHER at 25 yds. This is because the bullet is moving slower and exits the barrel later during recoil, when the muzzle is higher. Conversely, I would expect a heavier charge to hit lower at 25 yds. You've only shot up to 80 gr of powder so far. You may find that 90 or 100 will bring your group down enough at 25 yds to be in the bull.

Since you're on the paper, I would wait on scrapping the sights until you can shoot at 50 yds at least. Unless, that is you just don't like them.