Well after messing around all morning with this and not figuring it out, then finally bringing myself here and posting... I think I just solved my problem.
I resized a case (RCBS F.L. this is 30-06 btw) and measured at the shoulder. 0.440"
I seated a bullet without any powder. Shoulder measurement is now 0.445".
I believe I had the seating die setup incorrectly. I put a trimmed case into the shell holder and raised the case. I screwed in the seating die until it contacted the case and backed it off a full turn.
I seated a bullet and the shoulder measurement stayed 0.440"
The round seats fine now with powder.
So I believe I was swelling the cases with the seating die by having it screwed in too tight.