The Dual Information Bottleneck

Abstract

We present a new framework, the Dual Information Bottleneck (dualIB), which resolves some of the known drawbacks of the Information Bottleneck. We provide a theoretical analysis of the dualIB framework and solving for the structure of its solutions. To approach large scale problems, we present a novel variational formulation of the dualIB for Deep Neural Networks. In experiments on several data-sets, we compare it to a variational form of the IB.

Ravid Shwartz Ziv

AI Researcher

AI researcher at Meta MSL with a background in information theory and computational neuroscience, working on world models, memory, and compression. Former Assistant Professor and Faculty Fellow at NYU. My industry experience spans Wand AI, Intel, Google AI, and Wikipedia.