It sounds like you have a somewhat oversize bore. The bullet should be a sliding fit, at about .001 under bore diameter, but the plastic part should be a little over bore size. if the plastic part is falling through your barrel, it is too small for that gun. I would slug and mike that bore!! I believe the Big Bore Express folks make a slightly larger one (I am assuming a .50 cal. here) for the White rifles, which are larger by .002 or so. You might try some of these.

By the way, if you are getting 2" groups @ 100 yards from tumbling bullets, that's amazing!!

Keep yore powder dry!!