IMO, you should not be shooting in the direction of someone's house whether they are sitting outside or not. And not because they might call the cops. It only takes one flier, or an arrowpassthrough, or a freak ricochet and somebody being in the wrong place at the wrong time.