Couldn't you still try some rubber washers or some moleskin between the quiver bracket and the site bracket?

I love my soft lock and don't have that problem. I use the rubber gasket though and mount mine the way you describe minus the spacers.