Regardless of the brand or whatev, you need to get the other part out!

To answer your initial ? no your bow will not shoot "worse" without a stabilizer. You might shoot worse without it, but the bow wont. As long as you can shoot good enuff with it, then your bow will still push the arrows out the same.
Derek