I say all of the above. And you can try what I do. I wake up way early, well before sunrise, find me a likely spot and sit with my binoculars. I watch what the local game is doing and ask the questions, what, where, when and why. After awhile game movements start to make some sense. I enjoy watching as much as I enjoy actual hunting.
For me small game is where the fun is at. Jack Rabbit is my favorite, Dove a close second. I also enjoy hunting Varmints, mostly Fox or Yotes. I keep my eyes open when I'm hunting small game, I know where the Deer are.
Personally, Deer aren't that hard to hunt. I'm not much on Antlers and my freezer is always full. Basically my thinking is Deer are dumb, Fox and Yotes can be way smart. Older Hogs can also be way smart. I enjoy matching wits with them.
Hunting can take on some strange aspects. Some of the best times I've had was hunting Rattlers with my Dog. She'd point and I'd bag them early morning before they got very active.
I say don't get locked into that Deer and antlers thing, a lot of hunting to do besides Deer.