Efficient Training

When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models

Inheritune builds smaller language models by reusing useful layers and reducing attention collapse, while retaining or improving performance.