He simply didn't know what you were, on either occasion. I also think that you must've been downwind of him. Had he gotten a wiff of you, more than likely he wouldnt've hung around or certainly came back after you walked off. I likewise wouldn't call it aggresive behavior as they typically aren't aggresive this time of year. While still in velvet a buck will be cautious of confrontation and they aren't charged with testosterone like once they cast their velvet and begin getting ready for the fall.

The blowing was normal behavior and is used alot of times just as much as an attempt to startle something (just like the stomping motions) into moving as it is used to warn deer in the area that something is alert.

Have fun, they are very forgiving this time of year!
RA