I do 90-95% of my hunting before and after work. Even if I'll only have 15 minutes of light before I have to run for work, I'm there. Given how the very first and very last parts of the day are the very best, it's worked for me for years. I've killed a few deer after being in my spot no more than 2 or 3 minutes. I'm lucky enough to be able to hunt out my back door and have the freedom to call in late to work when I do score. With a full daytime office job, wife working most evenings, and young children,I have to squeeze in hunting time when I can. "Going to deer camp" and hunting all day are foreign concepts to me.
Sure you might spook deer going in late or leaving early. (And undoubtedly, you will.) But you certainly won't kill anything by stayinghome. My only problem with the before work hunts is that those early rises begin to take their toll pretty quickly. I'm almost glad to hear rain some mornings so I can sleep in.