Toughest to kill: mountain goat, without doubt!
Toughest to hunt: hard to say but I'd have to lean toward trophy whitetails.
My experience with "anything that can eat you if you miss" is that it ain't all it's cracked up to be, at least not in North America, Africa may be a different story...