That is a personal decision, you are right they will get bigger the following year if someone else doesn't get them after you let them walk. I will add something else here, are you hunting horns? If so let them walk, but if you are looking for meat and hunting public land you may as well take a spike or fork horn, they are better eating and stand a slim chance of making the season through.

The Tazman aka Martin Price
Founder and President of
Virginia Disabled Outdoorsmen Club