You will be fine with the altitude if you plan well. Regardless of what you hear "being in great shape" probably has nothing to do with whether or not the altitude will affect you, but it sure helps with the elk hunting! I have altitude problems and here is what works great for me:
1) Drink water and drink more water. Avoid caffeine or alcohol until you are well adjusted (usually 3-4 days)
2) Try and arrive early and stay lower for a night. The hardest time I have is when I fly and go from sea level to 10,000 ft in 12 hours. When I drive and stay down at 6,000-7000 ft the first night it helps. Also try not to do anything strenuous the first day you get to altitude
3) Take one of the herbal altitude products. I know of only two that work for sure: Altitude adjustment, and altitude advantage. I have used both and they really do help. Relatively inexpensive too.
4) Have antacids such as rolaids or tums. Altitude can make your blood slightly acid and taking antacids can help.
I used to take actual drugs (Diamox) for altitude, but it requires a prescription and has side effects. The altitude products have replaced that without the side effects and no prescription needed! I am not one who things herbs and cure anything, but these have both worked for me. I generally use altitude adjustment because it is made in Colorado and they are sold by 3 day packs. $5 each I think, and I have never needed more than 2 packs (6 days) so for $10 and shipping, I can be without altitude problems.
http://altitude-adjustment.com/