Be real sure that the load your shooting is grouping tight before you file that sight. Once you take metal off, you can not put it bacl In answer to your question,to bring your shots down, you have to (your rifle is hitting high) lower your rear sight, ot you have to get a taller front sight. So to raise your POI you would file the front sight, or raise the rear sight. Move the rear sight in the direction you want the group to travel. The opposite is true for the front sight.

Have you tried other sight pictures first? Also is this the distance you want the rifle sighted in at? I would move the box to 25 yard and shoot groups. It should shoot a pretty tight group at that distance. Then move back to 50 and do the same again. Again, the group should be tight. You should not be spraying all over that box. Once you are sure of the group and load, then try to figure the average distance you will have to shoot when hunting. I personally like 50 yards with iron sights, and 70 with scoped rifles. Then with a good shooting load, make the final adjustments to your sight.