No - be sure to take the ramrod out before you shoot it - unless you're making the transistion from archery to muzzleloading, then you can leave it in. Just kidding.
If you're talking about at the range, I leave mine out, since I put a T-handle on the end to make loading easier. But in the field, I always have it in. If yours is loose enough to slide out on its own, I'd make some modification to the thimble to make it fit a little tighter.

IM jaybe