I had the same experience. I shot roughly 20 times in one afternoon. I had at least 3 outright misfires and many, many hangfires - where the primer would pop and there would be a noticeable lag time before the powder was ignited.
It was extremely frustrating, so I ditched the whole idea. I don't know, maybe it works better in certain types of rifles and not others, depending on the breech plug, firing pin, etc.

I switched over to the Winchester Triple Seven primers- they work great and I get less of a crud ring with them than with the Federals I had been using. They are pricier, but it's not a huge deal unless you shoot a ton.

Personally, I'd ditch the Vari-flame and just use a regular 209 primer - you can get several types and see which gives the best accuracy and the least crud ring.