I've seen a fair amount of deer from the duck blind. Some of them were within bow range. We considered taking a bow to the blind, but never did.

I would suspect the only time the duck hunters will foul up your deer hunting will be with their ingress and egress. Other than that, the deer will keep to their same routine. I doubt the shooting will affect the deer unless the deer is near the blind when shooting commences.

Short answer is "go for it"