Look at it this way. If you had drawn and been spotted, they would have blown and alerted every deer within hearing distance.Versus not drawing, letting them walk off quietly and maybe having another deer meander by.
IMO you did the right thing.