Well there are things to consider here. 1. Friends 2.The Heard.
1. If you are a real friend, "as in grew up together" then you should try harder to get him to help you manage the heard.
2. If your does to buck ratio is good then there is no reason he cant shoot those bucks, basically could be cull bucks.
But if they are 1 to 2 year old bucks then i think he should pass. Simply friendship over bucks will cause the property to be pour hunting.
However bucks over friendship, you lose a hunting buddy.

So my advice is to talk with him. Better yet get with him in the field and film for him and help him decide yes or no.
If you guys dont manage the property you will decrease your hunting percentage and friendship along with it.
The 4 or 5 times he was asked to pass on those bucks, could have happened in a single season. Educate yourselves on how to manage the property. reason with him on limitations. Also, if others hunt there that dont have limitations on size of deer taken and he does thats not good management as well.
The rules have to apply to everyone including the owners. "Do as i say" "Not as i do" will not work for this situation.
Help your friend help you get it right. Dont just drop him after a couple of suggestions. Find out if he is hurting the heard or helping it before making such decisions.