I basically had the same experience with our middle son. He wanted to try shooting skeet with me when he was about that age. I got him one of those little Snakecharmer .410's, as that was what fit him at the time. He quit after a few rounds. He came around when he was older though, and we spent a lot of time shooting pistol matches together, until he got a driver's license and discovered girls.
I took a lesson from that, and didn't start our youngest out with anything bigger than a .22 until he was 12. He's now my deer hunting buddy and shoots his .243 quite well.