You need to do something, hunt more or find out what they are doing. If you have permission to hunt the area, get in there. If they don't....say something to them. If you are afraid of losing the ground by saying something....well, theres not much you can do, other than kill the deer before they get a chance. Who is telling you that they are killing the deer?