I've had very similar problems, and I am completely, 100% certain that the cause was me. More to the point, it was my poor form. I wasn't consistently squaring myself and so I was torquing the bow without realizing it, and the single cam Patriot is extremely easy to torque when compared with my old dual cam Martin Lynx. It's also an extremely accurate bow, so when an arrow doesn't go where I expect it to, I'm pretty sure I'm the cause. I've added a tied-on kisser to help me make sure my form is solid. So far, it's working.

As for "drop away time", I would certainly suggest that you switch to a drop away rest. I use a Drop Zone, and since I switched, I know I'll never go back. The performance is better and I find it much easier to tune.

Here's my setup:
03 Patriot @ 66lbs
389 gr Easton Axis 26.5" arrow with 100 gr Liberty Broadheads (2" overdraw to magnify my problems!)
TR Drop Zone