I've found that if my draw weight is too light its actually harder to hold steady. Increase your draw weight by 5 pounds and see what happens.

Another thing is proper draw length. Too short gives too much arm bend which means you're using the small muscles of your arm instead of the big muscles of your back and shoulders to keep the bow arm steady. Too long is bad too.

Also, do you use a stabilizer? The added weight of a stabilizer will keep your bow steadier while aiming. A heavy bow is easier to hold steady because of inertia.

Finally, do you use a peep sight? How large is it? Do you center the pin or the round pin gaurd? Make sure you center the peep sight perfectly every time. If your peep is too large it will affect accuracy at longer distances especially. I really like using a round pin guard and a peep that is just large enough to center the pin guard.

Also, how big is your 40 yard pin? Try a .019 instead of .029 diameter pin for your 40 yd pin.

Don't worry too much. You're still shooting very well at 40 yards IMO.