Personally, if you're thinking about getting to a ball small enough not to do a lot of damage to edible parts, you'd probably have to go all the way down to a .32 and I can't imagine it would be accurate;
In addition, multiple patches may allow a ball to slip/move forward, etc.

Onesuggestion would be to sight it in with a throttled back load like 40grns of powder, and just go for head shots, or dead center in the middle so no major edible parts are damaged....I do that witha .45cal I use for squirrels