There are several factors that will play into determing where to set up camp. The time of year, weather conditions, terrain, acclimation and type of hunt all play a part in that decision.
If you were just backpacking in I'd want to be as close as possible without setting up right on top of them. With horses I would camp at least a mile away if you plan on hiking into the hunting area daily and further if you're riding into your hunting area each day if possible. In fact if you're riding into the hunting area each day I'd want to strategically position myself to be able to access several different hunting areas from camp.
Terrain and accessing the hunting area will somewhat dictate how far you can be. You probably don't want to climb something physically draining every day to access an area.
Weather conditions and time of year will also determine if you should "play it safe" meaning you know you can get out if the weather turns real bad.
If you aren't acclimated real well to high elevations it would probably be good to sleep a little lower so your body can recover.