Assuming by "size" you mean spine.

First off, if you are under or over spined the arrow will generally have accuracy and consistancy issues.
If an arrow is severely under of over spined you could run the risk of the arrow cracking, splitting or possible breaking when the string is released. I've not seen this but in severe cases i would imagine it is possible. If that were to happen yes there could be a danger to yourself. If the shaft of the arrow were to become damaged and you didnt notice it and you tried to shoot it again it could split and wind up in your arm, hand etc

If an over or underspined arrow makes it off the bow undamaged there is no real danger to the animal other than the fact that it is about to be killed by said arrow if it is hit.