I take tons of kids out all season long. Primarily we hunt from ground blinds which will help cover movement.

The most important thing I've learned is when they say their ready to go, GO.
Some kids will go and set there all day and some will want to leave within the first five minutes.

One good activity I've found for the younger kids is to give them control of a video camera and let them shoot the squirells, turkeys, whatever comes out. Since your 4 year old will probably just be watching you take a deer, give him the chance to capture it on camera for you.

Good luck!