Yea, it could matter, because neck tension changes the more your brass is worked. It gets work hardened. In your average hunting rifle, though, I doubt you'd see much if any difference, but you might.

Personally, I wouldn't mix brass this way. I wouldn't want to loose track of how many times my brass was used since eventually it could fail leading to ruptures, splits and so forth.