So I noticed today my muzzle loader has a small knick if that's even what you'd call it in the crown. Just a slight rough spot you can feel with your nail. I don't know if it's what's called the crown or not. It's that part that looks stainless right on the end of the barrel where it's rounding out kind of. Close to where the end of the actual barrel is. Is this thing going to make it shoot way crappy groups now or is it fine and something I shouldn't worry about? A buddy looked at it and said he wouldn't worry it should shoot fine and a bullet would probably smooth it out in a shot or two.

Should I worry or am I most likely fine?

You should be fine as long as it is on the end of the muzzle.

There it is circled in red.

What do you guys think?

I doubt that will cause you any grief. If it bothers you, use one-inch squares of 600 grit emory paper and a round ball ( wood / large marble / round end of a screwdriver handle / etc. ) to polish it out.