They weren't there when you set it up, I suppose?

If you can clog the other holes.....I'm betting there's a product (other than gasoline.....which works fine around my property on bees in the ground) that you could toss in the hollow part. If you don't get ALL their escape routes, though.....you won't accomplish what you're after.