Sometimes it's kinda hard on the real young ones. But after a good look through some glass you should be able to see some little fuzzy bumps on his head. I also look at the glands on the back legs, not always, but the males seem to be darker. Lastly might be nice to let the youngest ones grow until you can tell, but if you can't be sure, don't shoot unless you have tags for both sexes. I shot a button this year too thinking it was a Doe, my bad, but I had a tag and the freezer was in need of meat. The season has since gotten better.

Also I must agree with, BowHuntingFool, the youngest bucks are pretty doppy and the older smarter ones let them walk first. I usually don't shoot the first one I see but wait for his buddies. At least that's how I would like to do things.