Model Compression and Efficient AI

Wed, 26 Aug 2026 00:00:00 +0000

Efficient AI is not one technique. A system can reduce the precision of its weights, reuse or remove redundant layers, change the structure of a linear transformation, or select a better internal representation for a downstream task. Each choice saves a different resource and risks losing a different capability.

My work connects three levels of compression:

Models and weights

You Had One Job treats mixed-precision quantization as a task-conditioned allocation problem. It uses hidden representations and output sensitivity to identify which transformer layers deserve more precision under a fixed bit budget.

When Attention Collapses / Inheritune studies redundant transformer layers and how useful layers can be reused to construct smaller language models. NdLinear replaces a flattened linear map with transformations along the dimensions of a tensor, reducing parameters when the structure of the task supports that factorization.

Representations

Layer by Layer shows that intermediate representations can outperform final-layer embeddings. Attention Sinks and Compression Valleys connects large residual-stream activations, attention sinks, and representational compression across model depth.

Representational compression is not automatically a smaller model. It explains which information is retained and where a model’s computation becomes concentrated. That understanding can guide practical choices about layers, embeddings, and downstream systems.

Inference

Efficient inference also depends on how a model generates. Min-p adapts a sampling cutoff to the model’s confidence. My essay Speculative decoding, from zero to DSpark explains how drafting, verification, and scheduling turn otherwise idle computation into higher serving throughput.

Explore selected systems or see all publications.

Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Time

Mon, 24 Aug 2026 03:33:39 +0000

XTC: Head-Aware Sampling by Excluding Top Choices

Mon, 24 Aug 2026 03:27:30 +0000

UAT-LITE: Inference-Time Uncertainty-Aware Attention for Pretrained Transformers

Tue, 03 Feb 2026 00:51:26 +0000

You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

Sun, 09 Nov 2025 19:58:24 +0000

The workshop version is identified in the arXiv record. The original preprint was released in November 2025; the latest revision was submitted in June 2026.

NdLinear: Preserving Multi-Dimensional Structure for Parameter-Efficient Neural Networks

Fri, 21 Mar 2025 17:52:44 +0000

Efficient AI | Ravid Shwartz-Ziv