I would change the glue you are using.

When I got into fletching my own, I was at the pro shop with handfuls of glue, cleaner, vanes, feathers, etc..... One of the older guys saw me and told me to loose the glue and cleaner. His recomendation was to use regular Windex to clean and degrease the shaft after glue removal and light sanding. As for the glue, he recomended a product called " Future Glue" . It is a super glue product and is available at Wally World and Kmart. It comes in a small bottle with a brush applicator. Sets in something like 15 seconds.

After the fletches are on, I use the brush to run a small bead down each edge and put a drop on the front.

Using this method I have yet to have a vane or feather come off except when I scrape them off with a knife.