If you are not occasionally (ocasionally defined as once every 3-5 years) finding sheds or dead carcasses of deer considerably larger than you regularly see, then you aren't looking hard enough or you are not looking in the right places.
Or....they just aren't there. You can not say, (sans some combination of naivete, ignorance, stupidity) otherwise, with any sense of accuracy. It's an opinion....based on a "whim".
If you are in fact ALWAYS (always as defined sometime during the season) seeing the largest deer your area has to offer each year, then your deer are really, really stupid.
See above.