For me, I'd say no but you need to make that decision yourself.
Big deer come from management of your land and if you keep killing the little ones then there won't be any big ones.
That is of course unless your neighbors are killing all the little ones too.

If you are worried about other peoples opinions, your hunting for the wrong reasons.