When you draw try pushing away from yourself at the same time as you draw it back.

Some people tend to "sky" the the bow when they have too much draw weight for them. Which means they aim it at the sky when drawing back.

Keep at it and build up those arms and shoulders. Gym isn't a bad idea either just do it right so you don't injure yourself.