I am going to guess that you're shooting a flintlock with pellets. And the best igntion comes when you prime the touch hole liner area. Your problem is the pellets. While the rifle is made to shoot pellets, they still have to ignite them.

Try this. With your small pan flask, dump about 10 grains of black powder down the barrel, then set your pellets on top of that. Primer the pan as normal and then give it a try.

In the future, ditch the pellets. Get some Goex 2f black powder and shoot that rifle like it was intended to be shot.