If your first shot was off a clean barrel that is not all that uncommon to have them hit outside of the group. That brought up a discussion a while back of shooting off a fowled barrel or a clean barrel. Everyone has their own methods.

It could also have been a wind gust. As you said the wind was strong and that is all it takes to push a projectile, especially a roundball off its course. Just keep making note of where you hit on a clean barrel and see if it happens each and every time.

Are you swabbing with alcohol first, then a dry patch, then pop a cap before you load for the first time? That is what I does and it seems to help most the time.