I don't think it's so much an issue of age more so the individual. This year while we were bear hunting, we had an individual who was about 50 come in and set up his deer cameras and mess with our baits even after the land owner had told him to stay out until after we were done bear hunting.
The guy is right that he has as much right to hunt there as you do, but by that I certainly don't think he has the right to push you off your areas or crowd your areas even. 160 acres is a good chunk of land. My family has land in central WI so I generally don't have to worry about that, but that land is 2 hours away so I do hunt a chunk near my house that I share with one other hunter and we just each kinda have our own side that we hunt and I've never had a problem. We're both in our 20's.
So no, you're not nuts. It just sounds like the guy is a dink.