You will lose more than arrow speed.You lose penetration and your accuracy is probably suffering,even if you don' t realize it.
I would definately say fix it.There can be all sourts of things causing it.Fletching contact,arrow spine,improper rest setting,cam lean,shooter torque.
Fletching contact would be my first quess.
Find you a good pro shop if you think it is over your head,if you are a do it yourself type person.Go to
www.eastonarchery.com and go to downloads and download the tuning guide.