Sounds like you're making the rubber too short. You only want enough tension on it to rotate the peep not stretch the rubber. But in all honesty this type of peep is old school. You can get a peep without an attached rubber and have it aligned at a bow shop by simply adding twists to the string. Or, you can have your D loop set so it is in position to align your peep when you are at full draw.
Another thing is that rubber can snap back and give a serious eye injury.