RE: arrow on draw
No, you are thinking of the launcher arm. The arrow holder is what holds the arrow centered on the shelf before you draw the bow. It may be too close to the riser and moving it may help.
How is the clearance once it is drawn? Is it ok, or is it still touching or really close? Your rest may need to be adjusted out some. Depends on how it shoots with fixed heads.
Another thing could be you just don't draw the bow steady enough. Maybe your draw length is not correct, or you have too much draw weight. Or you could just need more practice drawing smoothly. Some people put a lot of torque on the bow while drawing, this is why they have so much trouble with prong rests. Could be your grip as well.
If your arrow is really close to the riser when drawn you may need new arrows any way. Two causes for that could be hand torque or an improperly spined arrow.
However if the arrow is centered pretty well at full draw and you just have problems with it hitting the side while drawing my guess would be you are having trouble drawing the bow. Maybe reduce the draw weight a little?
Something else it could be is if you have a string loop there may be too much tension on the nock between the loops. You could try spreading them out slightly. When shooting off the string under the arrow it tends to put downward force on the shaft while drawing. With a string loop if the loop is too tight on the nock sometimes the arrow will swing to the side when you draw.
It's kind of hard to know without seeing you shoot the bow.
Paul