Hey Lil`Kid, there's really no danger in doing what you propose, but there's a practical limit.

A .32 ball in a fifty caliber bore would require triple patching with heavy denim for anywhere near a decent fit, and accuracy would not likely be great. Won't hurt to try for yourself though.

A .45 caliber ball (available in .440, .445, .451, .454 and .457) could be loaded with double patches and may shoot pretty well with a little experimentation with patch thickness and powder charges. Keep it light though or you will blow the patches. Something like 25 or 30 grains would likely do the job.

Or, as others have said, a 40 grain charge under a .495 ball should be sweet shooting at "head of bunny" distances and really won't tear one up to bad on a mid body shot.

The great fun of muzzle loaders is playing with this kind of stuff and seeing what you can make work for you.