That is a problem we have today....people DO judge hunting skill by rack size. Like it or not, unfortunately its a reality.
Guys in the midwest routinely put down 130-140" bucks year after year, with chances at true montsters. Does that automatically make them a better hunter than a guy in another area putting down 100" bucks? Not neccessarily. There are definitely alot of variables, but the #1 biggest variable is location. Either you have big bucks or you don't. Some areas a big buck is 100" and hard to come by, and in others that is a dink that most would never shoot and there are piles of them around.
Obviously if your neighbor shoots a 160", that deer was in your area and most likely on your land also. Just because he got him, doesn't mean he's a better hunter. I know guys that realistically suck at hunting and are total slobs, but they hunt in areas that hold big deer, so they end up killing some nice bucks regularly. They don't even come close to qualifying as a "good hunter", they they are legends in their own minds.
Location and luck are two huge factors in hunting. Obviously skill is another big one, but you can have all the skill in the world....and if your area doesn't have big bucks, your SOL.