I have a set of horns in velvet;I just dust them with a feather duster.
you can use a small brush like a soft tooth brush.that shouldn't hurt them.anything soft should get the dust off.
just not any hard brushing,although the velvet after it get hard is pretty tough.