Like everyone else has said, clean the chambers before you shoot 357s in it after firing some 38s.

As to accuracy, I've talked to several experienced revolver shooters about the subject. The general consensus is that firing a 38 out of a 357, or a 44 Special out of a 44 Magnum, a 45 Colt out of a 454 Casull, or 480 Ruger out of a 475 Linebaugh won't really affect accuracy noticeably. It's only once you try to fire relatively short 45 Colts or 454 Casulls out of a long 460 S&W chamber that you really start to see a decrease in accuracy, which I can personally attest to.