Ya' never stated how much powder/what powder you' re burnin' with that purple sabot. Those purple sabots walls & base are thin & will burn up -- causing accuracy problems with heavy powder charges. Plastic sabots hold up better with bigger variances like 44 cal sabot in to a 50 cal gun (+6). You are only using a (+4) by shooting 50s in a 54 cal. They will hold up with lighter powder charges, but not heavy ones.

Find you a sabot/bullet in 44 or 45 cal around 300 grains. If you' re using hot charges like 120+ grains & the new sabot/bullet doesn' t say on the cover that it will handle magnum loads, buy a pack of MMP sabots.... they will hold up.

Another option to correct bad accuracy is either too much ramrod pressure to seat a bullet or not enough. If you' re getting flyers, you' re pressure may be too great -- if you' re too low on the target, you may need more pressure.
Also, slow your practice shots in hot weather. I always take 2 rifles to shoot when hot outside. Hot muzzleloaders cause huge accuracy problems ... especially with sabots.