I think the most important factor is genetics. Without good genetics a buck will take years longer to develop a nice rack when compared to one with good genetics.
Look at it this way. Take two humans who decide to increase their muscle mass by lifting weights. When put on the same diet and same workout one is likely to put muscle on much quicker than the other. The reason being GOOD GENETICS. Even if you load the smaller guy up with protein he will have a tough time catching up to the guy with the good genetics.

Eddie