Im all for letting the small bucks grow but as stated elsewhere on this site quality deer managment is impossible on small tracts. You cant control what others shoot.
I kinda look at it like dont pass on amything you dont want someone else to shoot.
As for passing the small ones for me its mainly if you shoot the small one you wont be there if a big one does come along.