The 6s give you more pellets per load than the fives. The 5s have more but
not much knock down power. 6s are perfectly acceptable for use on easterns.
40 yards may be too far. try shooting less then 40 and see what happens.
also were the sixes high brass? There are several loads with the same shot size. not all perform the same.