I hunt a lot in archery season and rarely wear camo anymore.

My go-to shirt is a dark brown (polarfleece) w/cream colored lines dividing the brown into small squares. I like that shirt because it's soft, warm, and quiet (plus I own a couple of them - bought them on sale).
My "busted" rate is no higher with those shirts than with camo.

Same for other accessories. I have a couple camo gun stocks, but black or plain old walnut seems to make no difference.

I think what gives you away to deer (more than color) is reflectivity and motion. Set still with an orange jacket on and most deer will walk right past. I do prefer soft fabric, though.