If he isn't the only buck in the area then I wouldn't worry much about spooking him. He probably wasn't threatened and therefore shouldn't remember the incident himself. Besides theres probably other bucks in the area that didn't see you.