I always figured that it was from the big bucks fighting with the other bucks, it would make them look rough and scarred up. I also have thought that huge antlers rub on their backs, along with bigger bodied deer crossing through fences might have a harder time than a smaller bodied deer, causing them to lose some of that hair