Here is my 2 cents!

Broadheads tend to not like speed above a particular FPS, depending on setup. Throw in the inherent flight deviations that broadheads impart, throw in the possibility of having spine vs. draw-weight near the " red line;" I would also consider that a 3.5" vane/fletch may not be enough of a stabilizer for your setup.

It could be that the 3.5" vane is battling to keep your broadhead mounted shaft stable and on the mark, and is able to do so up to a certain yardage. Past a certain distance, the vane may start to lose the battle and the broadhead takes over and pulls the shaft off course.