I think hunter orange is only required during rifle deer season where I hunt, in Oklahoma. When you are hunting with a rifle, the rifle lets you shoot deer at such a stand-off range that whether you wear hunter orange or do not wear hunter orange should not be a dominating factor in your hunt. Pay attention to other details, and the hunter orange isn't going to blow your hunt: place yourself in the shadows, surround yourself with brush that breaks up your outline, move not at all or very slowly, cover your skin (white hands, white face) with some kind of material (gloves, face net) that impedes the reflection of light off your skin as you turn your head slowly, place yourself so the wind is blowing towards you from the direction you expect for the deer to appear from. If you DO NOT have these issues addressed, the hunter orange isn't going to make a difference -- these other factors will dominate. If you DO have these issues addressed, my bet is the deer isn't going to see you with the hunter orange on.

But as has already been said, the law calls for hunter orange so you need to wear it.