treb,
I'm in the same boat. I have a wife and two small kids (both boys, age 6 and 2). The older one is starting to spend time in the woods with me as well, and in fact crawled into his first treestand this past weekend with me.
We are 2 1/2 weeks into our season and I've only been able to get out on Saturdays and that has been limited too with the weather here (hard rain, and heavy winds). I set up stand in an awsome spot last weekend and am chomping at the bit to go hunt it, unfortunalty it will have to wait until next weekend as I'm taking the wife away for the weekend this weekend (this should help ease the blow of crunch time later in the season - brownie points).
While I do get the eye roll after she says go hunting whenever you want, she is overall great. She puts up with a lot of my crap during this time of the year, and while things may not always be "happy" around the house she never says anything to discourage the time I spend in the woods.
All and all, you're not alone!!