Language Models

When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models

Inheritune builds smaller language models by reusing useful layers and reducing attention collapse, while retaining or improving performance.

Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs

A decoding method that scales its token cutoff with the model's confidence, giving a simple way to control sampling as temperature changes.