Here is a link that should answer a lot of questions. Deer antlers grow over a period of 200 days. As long a they have shed by the beginning of this period there should be little effect. Once a deer rubs his velvet off he is no longer expending nutrition on growing them.
http://www.whitetailinstitute.com/in...s/mar04/8.html