Since your on the wrong team, I would sit there and enjoy just being in the woods. It's not about shooting deer. It's about enjoying quality time alone relaxing in a treestand watching squirrels.
It's tough when your time is limited. I always like to have 1/2 dozen places to go and hunt. I get bored if I don't see a deer with in the first half hour. Sounds like your in the right spot if there are a lot of acorns. Right now the bucks are gorgeing themselves on acorns. The rut should be starting real soon and they'll need the energy.
If this is your only spot, I would hunt it and on Sundays I would scout for new areas to hunt. Always talk deer in casual conversations. Don't worry about picking up new spots without scouting first. It's not hard to show up for an afternoon hunt and find a spot that looks good.
Good luck. I want to see you put some points on the board.