RE: 300WSM or 338 Win Mag for CO Hunting???
You may give the 7mm Rem. Mag. some consideration if you're thinking of shooting 165gr bullets in your .300wsm...
Maybe the .300wsm case capacity is affected by the 180gr bullets vs. the 165, but after looking into bullets BC for my .300win mag, I saw no reason to shoot 165gr over 180grs, so my belief is why shoot 165 out of a .300 win mag, or any .300, but perhaps there are better reasons for other cartridges to shoot the 165.
Of the 2 you suggested, I think for your needs I'd pick the .300 WSM over the .338 win mag, and both would do the job on an elk.