As long as the antlers were obtained legally, I guess I don't see an issue with it. I would say that I wouldn't be interested in purchasing them but some people do not have access to private land like I do and as has been mentioned, there are alot of applications such as art and furnishings as well as making a rattling set.
As far as buying them for the purposes of display, as a trophy, it may not be ethical, but people do that and I expect there is a market for it. In fact, I've seen stories on TV about catching poachers who literally take orders and go out and shoot a deer for someone.