Research Areas | Ravid Shwartz-Ziv

Memory, Personalization, and Continual Learning

Wed, 26 Aug 2026 00:00:00 +0000

AI systems need several kinds of memory. They must use information from the current context, retain useful knowledge over longer periods, adapt to new experience, and sometimes personalize behavior without damaging capabilities learned earlier. Treating all of these as one problem hides the tradeoffs that matter.

I am interested in four practical questions:

How can a fixed-size state retain the right information from a long context?
How can a model update one association without interfering with many others?
What should be stored in weights, external memory, or an agent’s working context?
How can a system learn continually while measuring what it preserves and what it forgets?

Memory under a fixed budget

My essay Editing a Compressed Memory explains one concrete version of the problem. Linear attention replaces a growing key-value cache with a fixed-size recurrent state. This makes memory and decoding costs independent of context length, but introduces interference when many associations share the same representation. Delta-style updates, decay, and gating provide progressively more selective ways to edit that state.

This form of in-context memory is different from personalization across sessions or continual learning in model weights. The common question is what information a bounded system should preserve, how it should update that information, and how we can evaluate the resulting behavior.

For agents, memory must also interact with planning and verification. Minitap studies a mobile-agent system that separates these responsibilities across specialized components, making failures easier to detect and recover from.

Read my writing and podcast conversations or discuss a collaboration.

Model Compression and Efficient AI

Wed, 26 Aug 2026 00:00:00 +0000

Efficient AI is not one technique. A system can reduce the precision of its weights, reuse or remove redundant layers, change the structure of a linear transformation, or select a better internal representation for a downstream task. Each choice saves a different resource and risks losing a different capability.

My work connects three levels of compression:

Models and weights

You Had One Job treats mixed-precision quantization as a task-conditioned allocation problem. It uses hidden representations and output sensitivity to identify which transformer layers deserve more precision under a fixed bit budget.

When Attention Collapses / Inheritune studies redundant transformer layers and how useful layers can be reused to construct smaller language models. NdLinear replaces a flattened linear map with transformations along the dimensions of a tensor, reducing parameters when the structure of the task supports that factorization.

Representations

Layer by Layer shows that intermediate representations can outperform final-layer embeddings. Attention Sinks and Compression Valleys connects large residual-stream activations, attention sinks, and representational compression across model depth.

Representational compression is not automatically a smaller model. It explains which information is retained and where a model’s computation becomes concentrated. That understanding can guide practical choices about layers, embeddings, and downstream systems.

Inference

Efficient inference also depends on how a model generates. Min-p adapts a sampling cutoff to the model’s confidence. My essay Speculative decoding, from zero to DSpark explains how drafting, verification, and scheduling turn otherwise idle computation into higher serving throughput.

Explore selected systems or see all publications.

World Models and Predictive Representations

Wed, 26 Aug 2026 00:00:00 +0000

World models learn useful structure about how an environment changes. The goal is not simply to generate a realistic next frame. A useful model should support prediction, planning, and action in a representation where the important dynamics are easier to learn and reason about.

My work in this area asks three related questions:

What should a model predict so that its representation captures the structure needed for downstream decisions?
How should we evaluate a world model beyond visual realism or a single probing task?
When an agent learns from imagined trajectories, how do errors in learned dynamics and rewards affect policy optimization?

On Training in Imagination directly studies policies trained on trajectories produced by learned dynamics and reward models. It analyzes how model error, regularity, sampling, and noisy rewards shape return estimates and optimization.

The broader program also includes joint-embedding predictive architectures. S-JEPA learns predictive speech representations with soft targets, while HP-JEPA studies latent prediction over graphs at multiple resolutions. These are representation-learning systems relevant to the world-model agenda; they are not presented as complete environment simulators.

My background in information theory and computational neuroscience shapes how I approach these systems: a representation should retain the information needed for prediction and action while exposing dynamics in a form that models can use efficiently.

See all publications or share a research idea.

Research Areas | Ravid Shwartz-Ziv

Memory, Personalization, and Continual Learning

Memory under a fixed budget

Model Compression and Efficient AI

Models and weights

Representations

Inference

World Models and Predictive Representations

Related research