I wouldn't worry much about it. If you have deer on the property, unless things change drastically the bucks will be there for the hunting season. One thing you don't want to do is visit the site too much and keep stinking it up with human scent. That may well be the reason you're not seeing anything but does and fawns. You didn't say how often you're going in there, but I would limit it to no more than once every 7-10 days.