Ok, now I understand. No you won't always see 4 deer the next year. Deer, both bucks and does are territorial. (Honestly you can't be sure they are not still there just because you are not seeing them, some of them are pretty good at not being seen. Many deer on the property I manage I never see except for game cameras.) Come fall the mother of the yearling bucks chase the young bucks out of the area, so they don't try to breed her. The 4 point or 6 point may move on also due to territoriality, or replace an alpha buck in an adjoining area. They will sometimes travel 5-10, or more miles to find suitable habitat.

When shooting does, it is a good practice IMO to harvest the mature doe as it will leave the button buck (usually a doe will give birth to twins, one male and one female) on the property, as the mother will no longer be there to push him out. This is applicable only where you are not trying to grow the herd. The female fawn will breed the next year anyway, usually giving birth to one offspring the first year she is bred.