Thanks to all who replied, especially iayohunter.
What was said has confirmed my suspicions about the bucks not needing the mineral as much, so they leave and try to claim the top breeding grounds.
I guess my next move is deeper in the woods and try to push the envelope without getting in his bedroom and pushing him off the farm.