First off, and it may seem like a stupid question, but did you glue the original inserts?

Yes, if you did glue them, chances are that pulling them out will damage the shaft.