Smacked him in the nose, and knocked him off the chair.

It doesn't sound like you were far from hurting him. Imagine what sort of blow it would have to be to do that to you?

I understand the kid was being a brat, but that's no reason to hurt him.