Looks like you have narrowed the problem down and on your way to a solution. I agree with lemoyne and try different weight bullets and different charges to get the best accuracy. My friends and I do not use Pyrodex as we have had very bad results with it. There are a lot of great powders out there to try. Triple Seven is a good choice. I personally love black myself. Triple Seven and black powder ignite when it gets cold and our seasons can get very cold. I deer hunted one morning when it was 17 below zero.
Good Luck and may that trophy buck be in your sights this fall!