Those bows (the patriot especially) have quite a bit of forward bias to start with compared to most other bows. Also, shorter bows ARE more sensative to putting larger sights on.
You may want to try a very light sight as pdq 5oh suggested, and a rearware facing stabilizer in addition to the normal one to help find YOUR balance for the bow.