If it is still firearm season and you didn't shoot at the deer in that piece I think you'll be ok. You may have pushed them out for a short time but once they get shot at again they'll remember that nice 4-5 acres where they lived without being shot at. I think it'll be fine as long as you avoid entering into this piece unless absolutely necessary.

I have two doe that I've jumped on my way to my stand about 15 times this year. I don't bother them as I'm just passing through. I think the deer use the same beds because like clockwork when I reach a certain spot on the trail they get up and bound away often times stopping and offering up a good shot. I've come to the conclusion it's best not to shoot these two deer. They are sort of my "live decoys" and I hope that bring me old mossy horns.

Enough rambleing. I think you'll be just fine.