nice rig i think that most of the reason you where missing so many fish wasn't due to the hook set but the fact that the bass didn't get to the hook yet. when bass are slugish and not real aggresive many times they will just grab the tail of the worm and slowly work there way up it. when they do this i try to give them a little more time to eat the bait and then i give a small pop with my rod tip and if i can feel the fish is there i will make my hook set. if i don't feel the fish but get a little tug and nothing else i will let it set for 3 maybe 4 seconds and try again. i've found many times that it will be just enough to get it out from the bass's mouth and normally the bass will come up and unload onto the bait.

also try to get the tip of the hook just out of the worm so that it will sink in more when you set the hook

good luck and have fun