Yes, you should let him walk. By doing so you would allow him a couple of more years to grow and, more importantly, spread his genetics into the herd.

That being said, if he walked in front of me I would have dropped him like a bad habit.