It won't damage your barrel, but if the action isn't properly blocked and bedded and free floated, it can change your POI. But agreed, that's dependent upon a pretty significant change in humidity.

I block and bed & free float all of mine, so even though they're in wood, I never have to worry about my POI shifting.