As Cayugad said, yes the clean barrel may shoot differently than a fouled barrel.
I've found this to be a norm. I don't like to foul my barrel(s) simply because I use mostly BP in all of them (Pyrodex in one percussion) and I may go an entire season without getting a shot. And I definately do not want to leave BP fouling in my barrel that long.
Because of this, I sight in with a clean barrel.
I didn't see were you said you swabbed between shots? Did you?
And actually, after re-reading your initial post - You're really not that far off left. And your height is basically identical to your second and third. So try it again and split the difference.