RE: Whats wrong with my setup?
A rest doesn't have to be set up just right to get arrows to group. In fact, the bow doesn't have to be tuned at all and it'll still group field points well. So long as the person shooting the bow is doing the same thing for every shot. The problem sounds to me like a shooting form issue. And that could very well be that the bow doesn't fit you well. This can make you shoot differently each time you draw the bow.
I see one guy mentioned locking the thumb behind your neck. This in itself can create torque. From experience I can also say that if you can get your thumb behind your neck then the drawlength of the bow is too long. Been there and done that---years ago.
Here's one I use for tuning a bow. I set my rest up on centershot suing a laser. I never move it. If my bow is shooting to one side or the other I adjust the weight of the bow. Who says that we must shoot a particular weight? Can you honestly tell the difference between 70 lbs and 67 lbs? If the bow shoots well at a certain weight then that's what I shoot. If it's too far off then I change to a different spined arrow.