My main concern with what you're saying is that there might be a possible short load going on there. Like Goatbrother was saying, this is dangerous. It sounds like you're feeling a false bottom caused by a crud ring.

Get a range ramrod that is six inches longer then your barrel. Put a good cleaning jag on the end of that. I use a jag that doubles as a cleaning and a loading jag. Save that ramrod in the rifle for hunting.

After you dump your powder and get your projectile started, push that down until you are positive you are on the powder charge. Like said, take a piece of masking tape and wrap it wound the range ramrod level to the end of the muzzle. This is a witness mark. All future loads need to meet this mark.

After you shoot, take a solution like Windex. Damp spray a patch and work that patch in short strokes down the bore of the rifle, cleaning fowling as you go. When you get near the bottom you will feel resistance. This is not always the breech of the rifle you are feeling. This can be a crud ring. So start putting a little more pressure on the ramrod and see if it does not start to move even further into the barrel. When you reach your breech plug, believe me you will feel the difference. You can push that wet patch down hard and it just stops suddenly. Also look at the ramrod. The witness mark should be under the muzzle of the barrel.

Pull that patch, flip it over, and do the whole process again. After that, put a dry patch on that jag and again, in short strokes work down that barrel to the breech plug. Feel that breech plug.

Now that you know the breech is swabbed and dried, dump your powder and push your projectile down. It should reach the same witness mark as before.

I think your accuracy is suffering because of inconsistent swabbing, and loading pressures. I would try 100 grains of Triple Se7en 2f or if you can find it, BlackHorn 209 and a 250 grain Shockwave. When you start to get a good group, then kick the charge up a little until the group opens up.

If you're shooting a scope, check the mounts and rings.