I wouldn't be too worried about it. I have tons of trail cam pictures of a group of nice sized bucks hanging out around of all things a permanent outhouse. My parents have an area on their property that they occasionally camp at that is only about 300 yards from their house. There are two outhouses and a picnic table. It doesn't appear to bother these bucks one bit.