That's what he is doing hunting for those hens by waiting them out on the sideline.
Call without a decoy and if he doesn't see a hen he might just be wandering about trying to locate you. If he sees anything he won't come as he is used to the hens coming to him when they see him.
Next time when you are out just listen if you can hear a hen calling. Every hen has a little bit of a difference in pitch on her. Try and match the call of that hen if you can the next time out.
This is what I would do if I was in your situation as trying to fool him with a call. Of course you can just ambush him but that's no fun.
Good luck, I can feel your predicament as I can relate to it a few seasons back when I wanted this particular Turkey. This Tom was a spoiled one. I saw at least 2 hens flew across the whole field just to be with him one morning.
One day I ended up following them to the nest site and lingered for a couple of hrs 150 yards away waiting silently and when I saw him by himself at the edge around 11:00am I started to call with a slate call which I though was my closest call to an immature hen I've heard before. There was no hesitation on his part. That was the most rewarding hunt that season and he turned out to be a beautiful bird with about 1 inch spurs. It took me almost 2 weeks of driving 1.5 hrs. to go hunt this bird almost every day,but hunting and learning experience was well worth it. Things I used to do I sometimes wonder but glad I can still do it.
Good luck Phil and make sure you take some good pictures when you get him.