But yet I passed on legal bucks, knowing and accepting that they very well may be taken by other hunters,
You passed on those legal bucks simply because you thought they did not meet your high standards and you wouldn't be able to brag about them to your buddies. You did not do it for the good of the herd or the future of the sport of hunting.