Switching to different kind of ammo may or may not change your impact point in any direction. It also might effect how the rifle groups at different yardages. The only way to really know is to try it and see what happens.

If it does change it should not change much. My biggest concern would be how the new ammo grouped.

Are you talking about having a deer load and a varmint load out of the same rifle? If you are I would set it up for the varmint load and then see where the heavier deer load ends up. Varmint hunting is usually at longer ranges and more precise than big game hunting. Not that I wouldn't strive for accuracy in big game load. But if it wasn't perfect I don't think it would keep you from getting game at average distances like 100-200 yards.

If your just talking about finding a new load to use I would try several until you find the one that groups the best and then re-zero your scope when you find something you are happy with.

Paul