It's really up to you!

Biologically the fawns are yearlings and they are weaned and can live without Mom. If they were button bucks, they will be chased off shortly.

If you would feel bad or guilty, then you did the right thing