Model Compression

Model Compression and Efficient AI

Ravid Shwartz Ziv's research on model compression, task-aware quantization, efficient representations, and reducing AI memory and computation.

When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models

Inheritune builds smaller language models by reusing useful layers and reducing attention collapse, while retaining or improving performance.

You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

Compressing an LLM for the task it actually needs to perform, allocating precision to the layers that matter.

NdLinear: Preserving Multi-Dimensional Structure for Parameter-Efficient Neural Networks

Reducing parameter costs with linear transformations that preserve the structure of multidimensional inputs.