not adjusting the scope, my groups were 5" higher. Is this what should have happened?
Any time you change bullets you must sight your rifle in again. That bullet could have impacted left, right, up, down or a combination of. No two bullets impact the same (usually).
After you sight your rifle back in again with the new bullets then the trajectories will be similar.