I having this issue as well.I generally fish for larger fish such as northerns and catfish which hit pretty aggressively and have larger mouths to get the lure/bait into.

This year Ive started doing a lot of walleye fishing and its been very aggrevating for me.The small ones will hit the lure like a truck right now, theres no doubt you have one on.The larger ones seem to hit more delicatley.Last weekend I was in a fishing tourny for walleyes and right at dark had a hit, except I thought it was a snag, instead of setting the hook I pulled back slowly to get my lure out of what I thought were rocks.About a 5 lb walleye was on the end of the line, he actually stayed on till I got him to the dock where he threw the hook, had I made a good hook set I would have taken 1st place.

later that night on the other side of the lake a much larger fish, probably a northern or catfish sucked in the plastic worm I was using and kept swimming.Again I thought it was a snag and again I had the fish on for just a few seconds before he threw the hook.

The next night I switched gear completely, went with smaller line, smaller rod, and a different reel which was much more sensitive to every bump in the water.I reevaluated my lure set up and redid my stinger hooks i was using keeping the line tighter on them and hooking them differently.Unfortunately the fish kinda shut off but the ones I did catch i knew without a doubt they were a fish and not rock or log.Without the slack in my sting hook line my hook set was much more effective and I landed the ones I did get to bite.

I lost the tournament but in three days I figured out a lot about getting less agressive fish to stay on my line as opposed to missing half of them.