RE: How old should the little guy be?
I take him at 6, but probably not a 4 (but it all depends on their maturity).
Can they sit quietly? Do they have patience?
In any case, I would defintely take them as soon as they are ready (whatever that age may be). You and your wife are the best judges to that.
On a related note, I think all kids should be required to go hunting even if only to observe. After they do that, then they should be required to visit a slaughter house if only to observe, so they can see where their hamburgers and chicken nuggets come from and how they are processed.
If this was all required, I think we would not be losing the younger hunter.