Ok this is how I handle it.

I say no, if I let you, I have to let everyone. Yes I am being rude, but I am being rude to everyone who ask.

My land, and more people I let hunt, the less deer I will see. I do not say I am sorry.

Once you established the "rude" factor you will be ask less and less.