Regularizing intermediate representations to reduce collapse and support sequential reasoning.
A decoding method that scales its token cutoff with the model's confidence, giving a simple way to control sampling as temperature changes.
We provide an information-theoretic analysis of VICReg, deriving theoretical foundations for deterministic networks and introducing new SSL methods based on these insights.
We show that carefully tuning standard deep learning components can achieve state-of-the-art performance on class-imbalanced datasets without specialized techniques.
We present a comprehensive review of self-supervised learning through the lens of information theory, introducing a unified framework that encompasses existing approaches and highlighting the interplay between compression and information preservation in deep neural networks.