Move the rest to the right. The bow string should split the limbs and cover the rest and sight pins. Sounds like your setup is set to shoot the arrows leftward instead of center.
Shooting through paper may provide valuable information regarding the direction the arrow is leaving the bow.
Finger shooting can affect the arrow flight. If you are using a release, there should be no horizontal influence upon the arrow unless your setup is incorrect.