RE: Speed or KE?
I've never killed a deer purely on the basis of my bow's Kinetic Energy output - and I hunted with a high KE setup for a long time.
I have, however, killed a deer purely on the basis of my bow's outrageous speed. 38 yds in 2007 - a shot that I'd have never taken at 270 fps. But at 350, even though I wasn't exactly sure of the yardage - I didn't hesitate to turn it loose.
I had two passthroughs on pretty solid bucks last year using arrows that most of you guys wouldn't say were fit for shooting chipmunks. 290 grains @ 350 fps.
Like Dan said - balance is probably the right answer for most people - but you can go extreme in either direction - and have little or no problems filling your tags.
Getting the shot is the hard part. The arrow you're using is rarely the determining factor of your ultimate success.