That is not really impossible. Although it does take a lot of skill and knowledge of your weapon and load to make that shot. But during the Revolutionary War, one of the greatest threats to the British Officers were American Snipers. While the British army was armed with the Standard Brown Bess Musket which in battle line formation is a very formidable weapon... many of the roaming bands of American free men or militiamen were armed with rifled barrel roundball rifles. And some of them often made man size shots at that 300 yard distance. Most of our pioneer militiamen were able to do 100 yard shots all day long. Now that does not sound like much, but up against a smoothbore musket, that is a real advantage.
Now I am not saying that a balloon is the size of a man. But many people with the correct rifle, ammo, powder and PRACTICE can train themselves to make some amazing shots.