When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models
Abstract
We study attention collapse in decoder-only language models: deeper attention layers can become redundant. Inheritune reuses the useful early layers of a larger model, then trains and expands the smaller model. Experiments show that these models can match or improve on larger models with fewer layers.
Previously titled Inheritune: Training Smaller Yet More Attentive Language Models. First posted on arXiv on April 12, 2024; the journal version was published in TMLR in February 2026. The arXiv version was updated on February 16, 2026.
Ravid Shwartz Ziv
AI Researcher
AI researcher at Meta MSL with a background in information theory and computational neuroscience, working on world models, memory, and compression. Former Assistant Professor and Faculty Fellow at NYU. My industry experience spans Wand AI, Intel, Google AI, and Wikipedia.