Outdoor...It sounds like you are not cleaning between EACH shot....
I always do...The barrel has to be conditioned the same each shot
to get consistant grioups...

As far as different weight bullets grouping differently, because of the lower
velocities in a muzzle loader this is more evident than in a centerfire...It would be uncommon for a 250 to group in the same group as a 300...

How much load tuning have you done??? Are you just using the sabots that
came with the bullets?? Sabots can make a huge difference..As well as powder charges...I'd pick a bullet, start at 90 grains and work up to 120 before I discounted that particular bullet...Even if that bullet doesn't group well, you could even go to say a MMP-24 and start the process all over again...Just like reloading, or tuning a bow...I bet your broadheads hit in the same group as your field points, because you have spent a lot of time tuning your rest so they will...Muzzle loaders are the same way....Many folks are happy with 2 inch groups (with a scope) at 100 yards and that's fine, but they will do better than that...