It's amazing to me how we've been programmed by television shows and product hucksters to believe that unless we have hundreds of dollars worth of scent-block products and get into our blinds at 4 a.m., we're screwed and should just stay home.
Do whatever you want, go out whenever you can. As many others have pointed out, deer have been taken at all hours of the day. As a bonus, you're out there at the prettiest part of the day when you hunt late morning and early afternoon.