I would seriously doubt a loose ferrule would cause much of an accuracy issue. Just align them and tighten them back up. Think what you are seeing is normal and allows that ferrule to pivot a little - probably saves the screw threads if folks aren't careful when reinserting the rod.

Yes,I shoot with ramrod in place as that is where it will be while hunting.