Adam, I've done the same thing before and in my experience it shouldn't bother a thing. I've noticed that turkeys have a pretty short attention span. I've called turkeys in that we had just spooked out of a field. They had forgotten that we had just spooked them not 5 or so minutes before.

Same thing with roosted birds. I've busted birds outa the roost before just to have them fly down and with a little calling they start to come my way. They can be so love sick that they drop there basic survival instincts, just like bucks during the rut.
I think when theres love in the air, the turkeys don't care.