Yeah...it totally depends on the caliber. All calibers fly at different trajectories...making their point of impact different at yardages.
I look at where I'm hunting and the distance of my average shots.
At our farm in WV...we have guys that set their guns up to shoot 3" high at 100 yards. Probably 5% of our deer are killed over 100yards, but they zero their guns for around 250. Never made any sense to me that they would rather hold 3" low on 95% of the shots than hold 3" high on 5% of the shots. LOL