Last year I left a load in for a year because my breech plug tool broke and I couldn't get it out. The next season i went to sight it in and it just made a thump and lobbed out of the barrel. The bullet landed maybe 20 ft in front of me then it bounced somewhere i couldn't find it.
So no i wouldn't leave a bullet in just because i wouldn't when that happening when a deer walks by.