I always appreciate people thinking about what is on the other side of the target, as I have been hit with spent rounds too. (Hmm, probably many here have!) The other side to this, using CB caps which are remarkably quiet in a rifle barrel, is that you can choose the downrange path. On those relatively few occasions where I have popped one into my backyard, I have an out building and a good sized red oak as back up for any skip. If you can fire from a room, quieter yet. As a kid I used to be able to find BB caps, a primer based ball round, that were excellent at yard ranges. Would even go through a rabbit. But have not seen any lately.

The other hazard in populated areas is someone blundering into your line of sight just as you pull the trigger. A work partner not too long ago took a round from a german made air rifle and ended up in the ICU. So if there is any concern like that in my mind, I use a spotter.