That's not the only white oak with acorns in the area, they have others...See if you can find the white oak closest to their bedding area, that will be the first one they hit when they get up in the afternoon...

They are still in the area, just not out in the open eating alfalfa...