Ok, remember Robin Hood? The whole thing was killing the crown's deer is bad. They're his, because they taste so good. You want venison? Possibly more expensive than lobster. So it must be better.
Now, reality. Deer is a great meat. Not my favorite (give me king crab), but it heavily depends upon conditions (as stated above). The backstraps are always amazing. With enough treatment even the ribs (normally considered "dog food") are edible. Everything else is great if cooked properly.