In my experience deer will come back if you do not continue to spook them. Give him a week. On the other hand now may be the best time to get done what ever work you were going to do to avoid spooking him in the future.