I try to be considerate ,
and in your shoes I'd try harder to contact the neighbor and come up with a reasonable system for when both of us would be out . Leave your phone number in his mailbox and see if he calls . The odds are that he's wondering the same thing .