Bacteria begins to grow at temperatures above 41 degrees and below 140 degrees - called the "danger zone" in the world of food safety. If meat remains in the danger zone for longer than 2 hours, you begin to run the risk of dangerous level of bacteria growing on the meat, which could lead to illness. While illness isn't always the result, the quality of the meat is almost always impacted.
I butcher my own deer. If its above 40, I quarter the deer and put it in my fridge, lettting it age for 5 days before butchering it into steaks, roast, etc.