Well thats for you to decide. I personally enjoy hunting and I leave my standards where they are. If I shoot a smaller buck, it's just that much less likely that I will see good ones next year. If you plan on hunting the property for years to come, I say let the little guys go, It will be worth it in the next years, when you see more mature deer.
Like i said though, it is up to you, if you are really wanting a mulie, then take one. Every hunter has a little different standards. Good luck, and if you do take one, show us some pictures.