It's called integrity.You made a generous invitation and explained the terms.It is really a yes or no thank you response.If they were to say no thank you because they didn't feel they had the restraint not to shoot a buck then fine.
Your land,or your access,or your scouting and effort,your rules.That is not selfish.Someones friendship and trust is worth a lot more than a deer!