I live in the mountains in VA, our deer are typically not big (rack or body). I've hunted over 30yrs, the largest body deer I've personally ever killed is 130lbs dressed. I'm not saying they don't get bigger....but the avg for a mature buck in this area is about 120lbs dressed. We have practiced QDM on our farm in WV for 12yrs. The numbers of nicer bucks have jumped way up....and we've killed a few pretty nice ones, but the largest body weight was 145 dressed (in 20+ yrs of 10+ guys hunting this farm).
I would say mostly its the strain of deer, in combination with nutrition. Even the deer we kill that are 4.5 or 5.5 don't have the body weight you would think they should. Last year we killed a 5.5 that weighed 115 dressed.
There are areas in VA where the deer get alot bigger - both body and rack, but compared to other states (north or midwest) they are not very big. Compared to more southern states... ours are whoppers. LOL.