I'm no expert, and those who are will probably disagree with me, But I just don't see him getting a much fuller rack with age. There are plenty of bucks out there that'll grow an incredibly wide spread with thick antlers, but still have pretty short points or lack points on the inside altogether.
As others have pointed out, he'd certainly gain body mass with age if you let him walk. The ends of his antlers might even get longer and have a mule deer like look to them. Still, I think his genes are such that you're only going to see marginal improvement in his rack. I wouldn't feel bad about taking him if he presents the opportunity.