The wind you talk about may be considered a calm day in many places, but considered breezy in others. I doubt that much wind would bother the deer too much, but swirling winds do seem to make them nervous.

For shots under 100 yds I would adjust my POA only a couple inches, and only if I had to shoot during a steady wind. I'd first try to wait for a lull.

I love to stillhunt during heavy (treestand rocking, limb breaking) winds. The deer will be bedded in predictable spots and if your're careful you can get on them before they see you.