Hyawatha,
Those " Pig Points" should actually make it easier to pull your Terminators out of the target.....they have that slightly oversized shape to punch a slightly larger path than the diameter of your shaft preventing that annoying bonding of foam to the shaft which makes them hard to pull out.

Regardless of the diameter point you are using........your inserts should NOT be pulling out. Sounds to me like they were not glued in properly. slightly rough up the inside of the shaft itself, clean it with alcohol , let dry, and then glue the inserts in with something like Goat Tuff. They won' t come out anytime soon.