If he took off just down the way from you, I don't think he's trying to harass you. Probably doesn't even know you're there. I own an ultralight aircraft as well. I've never been shot at, but if I were, I'd be on the phone to the authorities real quick. Anyways, I know you were just joking, but I've heard of people shooting at ultralights. You don't just go to jail, you go to "federal pound me in the ass prison". Ever seen Office Space?
Anyways, my aircraft is a powered paraglider. It flies about 25 mph tops and is an absolutely GREAT way to go scouting for your next hunt!