RE: Is it my shooting or something else?....
if you're using loose powder, 100 grains pyrodex is 20 grains to much.
You can also try free floating your barrel.
Also can try replacing your breech plug, these wear out over time.
Cabelas sells a better breech plug thats made by CVA.