Bucks will disperse then, too. Move off to find their own territory. The following year, you may have a different set of 2.5 year olds than the 1.5s you have now. Plus, there's a lot of bucks moving back and forth during the rut.
Unless, you have a high fence, you don't know what deer will stick around and what new deer may move in.
Plus, 1.5 is way too young to tell. Even 2.5yo. I think you need to wait until 3.5 to get a sense of it. Sometimes bucks will have big antler jumps at 4.5 or 5.5 even.
I think culling bucks is stupid and it's degrading to the game animal.