Information Compression

Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin

Connecting attention sinks to representation compression to explain how information changes across LLM layers.

From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning

Comparing how language models and humans balance compact representations with semantic detail.