It's more dangerous to cast lead bullets. Once you get the lead vapors going you can get seriously lead poisoned from breathing it.
You should be able to find that size lead pellet in the deer. I can usually find most of the shot in the birds I shoot and those are pretty small.
Don't worry about it.