Ravid Shwartz Ziv's research on model compression, task-aware quantization, efficient representations, and reducing AI memory and computation.
Finding useful embeddings inside language models. We study why intermediate layers can outperform the final layer and how to identify them using information, geometry, and downstream evaluation.
Comparing how language models and humans balance compact representations with semantic detail.
Adjusting dropout during inference to estimate uncertainty without additional labels or training.
We provide an information-theoretic analysis of VICReg, deriving theoretical foundations for deterministic networks and introducing new SSL methods based on these insights.
We present a comprehensive review of self-supervised learning through the lens of information theory, introducing a unified framework that encompasses existing approaches and highlighting the interplay between compression and information preservation in deep neural networks.