I'm also a part-timer when it comes to practicing (can I use that word here?) with my bow. I consider myself to be a very good shot, despite only shooting for an hour or so about twice a week (on average) from a month prior to season through the end of November. It seems that I'm about middle-of-the-road in my amount of practicing. I have struggled greatly with target panic in recent years. The cure was due in no small part to simply limitting my shooting frequency and duration. If I shot 24/7/365, I would likely be at a MUCH lower level of skill than I am at today. Moderation has been the answer for me.

Also, this may not be the norm, but I actually feel much more focused and poised to make a perfect shot on an actual animal than I am at a target. This shoots a bit of a hole in the arguement "if you can't make a good shot on a target, then how can you hit the mark on a deer?" That's not to say I'm a poor shot at a target or that I condone hunting deer without the ability to hit broad sides of barns.

One more thing:
I disagree with the premise that one must have to have a love affair with shooting a bow in order to "mess with bowhunting". I enjoy shooting my bow. It's not the end-all for me, though. However, I LOVE bowhunting. Bowhunting for deer is what I love to do most of all. When I'm bowhunting I'm actually doing very little shooting. Bowhunting is not all about the shot. The shot is extremely important, and should not be taken lightly, but in the grand scheme of the hunt, the shot itself (IMHO) contributes only a mere instant to an experience that lasts an entire season. The memories last a lifetime. Try to gain that from shooting at targets in your back yard.