I'm not sure that camo makes a huge difference, heck a plaid shirt will look almost the same as a camo one to an animal as colorblind as a whitetail deer. Another facet that might be well worth considering is that most people do not wear camo clothing day in and day out, especially outside the hunting seasons. As alluded to above, depending on the type of hunting you do, scent control may be of even more concern and part of a dedicated regimen of scent control is to not wear your hunting clothes other then when hunting.
The foreign smells clothing picks up during a typical day around town are smells that deer are not so familiar with and tend to shy away from at all costs. That in mind, buying camo clothing to use
exclusively when hunting might be a good way to also become a hunter that's more scent-free and adds success in this manner, rather then giving the "credit" to the camo patterns alone.