I had a simmons 44 mag rifle scope do the same thing... would hit dead on at 75 yard... and the more shots I would do it crawled down the paper. Apparently it could not take the recoil. Changed the scope and all is fine.

On another thought... besure you have a hot enough primer to burn all the powder...I was using 150 grains when I first started muzzleloading until I was once sighting it in at dusk and I watched little balls of fire bouncing on the ground in front of me after the shot. If it is not burning completely in the barrel you are not gaining any benefit from the additional powder... and would be experiencing allot more fowling... If it is not completely burning you should be able to see some of the residual in the barrel should look like little balls/clumps. I now shoot 100 grn with CCI reloading primers and have a very light dusting in the barrel after the shot.