The general rule of thumb is that for every five pounds of draw weight you go down, you will lose 10 fps...

Correspondingly, that means you could conceivably pick up approximately 4 fps, if I'm doing my math right... You'd NEVER see the difference, and only you could determine if it was worth it as far as ease-of-draw, ensuring you weren't overbowed, etc.

I'm not even that sure it would change your point of impact at all.