Does it really matter if your soil is one or the other? Also are there some type of plants that cant grow in one, but can grow in the other? My last question is if your soil is constantly wet like mud, what plants do you recommend.

i would say it does matter what kind of soil you have

It makes a big difference, especially if your soil is not well drained.

Sand= well drained, going to dry very quickly, hard to get a lot to grow
Clay= holds water well, not well drained