Im in the antlers off camp myself. During the peak of the rut, I've had better response that way, but I certainly do know that bucks will respond to a buck decoy, sometimes violently.
Personally, I would NOT have touched the decoy with bare hands. A deers sense of smell is 1000s of times more sensitive than ours. If the wind is right, he may catch a whiff of whatever may have been on your hands. Also, NEVER spray the decoy. You now have a decoy that smells like a doe in estrus. Theres more to a decoy than just setting it out. Proper placement plays a role.