Check your scope. It can be loose or damaged.
I have a Hasting rifled slug barrel with cantalever scope mount and it shoots great.
I shoot the Winchester standard velocity 2 3/4" BRI sabots.
At 25 yards you should be putting them in the same hole.