It's been said that a rutting buck may stay with the same doe for upto 3 days, plus like has been said, bucks travel. I shot a buck that my buddy had pictures of 1.5 miles from where I shot him, and the rut hadn't even started yet.
They travel, that's a fact.