Branches...and entire trees...come down all the time naturally, so I highly doubt that a few branches on the ground will alarm them. If they're full of your scent from handling them, then they probably will be more alert. I watched a small 5-pointer sniff the tree where I had placed my hands while setting up my tree climber, so it doesn't take much for them to smell you.
If you're setting up a stand (e.g. ladder or hanger) that you intend to leave there for a while, the best time to do it is in a rain storm IMO. The rain will both dampen the noise as well as wash away your scent. When it rains, I take to the woods to make any last minute preparations that I didn't get to earlier in the year.