I think that this is a hard question, especially because each time you go out hunting you are presented with a unique animal, environment, and opportunity. For example hunting whitetail deer over a clover field on private ranch in Kansas, I would say is much easier than stocking a big buck on public land through an evergreen forest in Saskatchewan. The same could be said for a black bear hunt over bait in Ontario vs. driving out laurel patches in PA or NY. And the same for other species. Not only does the quarry vary the difficulty, but also the hunting area and method used to take the game.
To give my experience, I have hunted moose, deer, black bear, turkeys, and small game. I will limit my thoughts to North American Game animals.
However, if we talk in generalities based on typical environment, animal behaviors, and typical traditional hunting methods. I would say that you need to cross out the species that can be easily located or brought in to the hunter via calling, such as coyotes, wolves, moose, elk, turkey, and whitetails.
Also I think we would also need to cross out those species that are typically hunted over bait or based on their behavior frequent key food plots typically, coyotes, deer, and black bear.
I will also eliminate those animals that are typically hunted by chasing with dogs: upland birds, turkey, deer, black bear, and cougar.
Also eliminate those that herd, by which they often feel safety in numbers and a large group of animals are easier to spot and stalk: caribou, elk, musk ox, bison, turkey, and pronghorn (to some extent).
This leaves us with: Grizzly Bears, Sheep, and Mountain Goats.
Grizzlies are the most dangerous, but probably not the most difficult.
Based on my speculation I would say that the shear remotness and ruggedness of the mountain ranges where goats and sheep are found. And the physical stamina and strength that is needed to reach these areas combined with the traditional +300 yard shots that are often required to be made across a mountain valley with winds swirling and drastic elevation changes, (I would assume) make these the most difficult species to hunt.
Although it would be a great boost to my ego, if a sheep hunter disagreed and strongly protested that sheep are much easier to hunt than whitetails.
