The Dual Information Bottleneck

Image credit: Unsplash

Abstract

We present a new framework, the Dual Information Bottleneck (dualIB), which resolves some of the known drawbacks of the Information Bottleneck. We provide a theoretical analysis of the dualIB framework and solving for the structure of its solutions. To approach large scale problems, we present a novel variational formulation of the dualIB for Deep Neural Networks. In experiments on several data-sets, we compare it to a variational form of the IB.

Ravid Shwartz-Ziv
Ravid Shwartz-Ziv
Research scientist & PhD Student

Related