Inheritune builds smaller language models by reusing useful layers and reducing attention collapse, while retaining or improving performance.
Regularizing intermediate representations to reduce collapse and support sequential reasoning.
We provide an information-theoretic analysis of VICReg, deriving theoretical foundations for deterministic networks and introducing new SSL methods based on these insights.
We show that carefully tuning standard deep learning components can achieve state-of-the-art performance on class-imbalanced datasets without specialized techniques.
We present a comprehensive review of self-supervised learning through the lens of information theory, introducing a unified framework that encompasses existing approaches and highlighting the interplay between compression and information preservation in deep neural networks.