I suggest wearing at least a blaze (hunter) orange or florescent pink hat (hat and vest would be better) while on your land during deer season if they get access to adjoining land. We don't want them to mistake you for a deer dear.
Even if they don't hunt on your property, I strongly suggest getting an umbrella policy. These policies typically are not that much (might be equal to the cost of one or two hours for an attorney). Be sure you know what the policy will and will not cover before you make a decision on a policy.
You mentioned this land is currently vacant. Having a "No Trespassing or Hunting Without Permission" sign MIGHT give you more legal protection if something did happen. I would also put down in writing with these guys exactly what permission you have given them and not given them (for exclusions state 'including but not limited to') and have them sign it. Make sure it states excluding hunting rights.
With vacant land, you may want to look at a CCTV system if you have electricity on the property. The cost of CCTV systems are not that bad. I suggest at least one good quality camera that has infrared. The other cameras would not have to be as good of quality. It would also give you an opportunity to watch the wildlife which I think is always nice. If there is no electricity, there are small battery operated digital cameras that look like motion sensors for an alarm system. The photos are stored on a removable memory card.