Thank you all. I believe I found the problems.
First, I was not aiming with the open sight properly. I am still getting used to an open sight, and this was the key reason I was all over the place. I was not aiming with the sight properly. I tried a different way of aiming, and it worked.
Second, the grains and powder brand. I used 70 grains today successfully. It worked well. I also changed the pyrodex powder itself to a different brand, also worked well.
Finally, the patch and ball. I was forced to change the ball size to 0.495. The store was out of .490 size. I also changed my patch size to .010 patches. I had better success with these, for some reason.
I got a 1" shot group at 100 yards with these changes and a little experimentation. Attaching a picture with my thumb as a reference for size.
Thanks again for the help, Ill let you all know if any more questions come up.
Jared