I think it all depends on the person. The only time coffee has messed up one of my shots is when I cannot put my thermos down fast enough to grab the rifle.
I can drink a cup of coffee laying in bed and getting ready for sleep but for some folks the caffeine shock may mess with them.
I've seem the same thing with folks who usually don't chew tobacco. They decide hunting week is the week they will get some skoal or chew. Next thing you know they are ready to head back in because they feel sick and dizzy.
Drugs affect people in different ways, so yea some folks shouldn't drink it and in some folks it makes no difference.