First off, Antlers in velvet do have nerves. Velvet is a soft living tissue that is sensative to the touch for deer. When velvet starts to come off it actually bleeds and I do think it can irritate the buck as the living tissue dies and hardens- (similar to a scab).

I have watched bucks, while loosing velvet rub antlers into thick brush - It sure seemed to me like they were trying to get off the velvet.

Second - Antler color has nothing to do with what trees they rub. 90% of the rubbing is done at the bases and brow tines above the forehead glands. They rarely rub with the sides of the antler, or the back, or even the forks between g1,g2 and g3 tines. Yet antler color is pretty even throughout the antler. The bases will appear darker if dark bark gets stuck in the bumbs. If anything the highpoints of the base will become very light as they get polished - no matter what color tree he rubs on.

I have seen pen raised bucks with perfect colored antlers - that never rubed a tree in there life. Some where dark and some where light.