In my experience (I'm 24) it goes like this: When you start hunting (at whatever age) you get nervous all the time. You miss deer, and unfortunately wound them. It's a learning experience. Eventually, you get a few deer and you gain some confidence. Once you have that confidence that you can make the shot, you stop making those rookie mistakes like rushing the shot, aiming at the wrong place, drawing at the wrong time, etc.

Once the shot process becomes something you can do without thinking, you'll stop having that panic feeling because you know that you can do what it takes to get that deer.