TL;DR
A phenomenon in large language models where the attention mechanism allocates a massive amount of focus to the first few tokens in a sequence, regardless of their semantic meaning.
This behavior emerges because the softmax operation in the attention mechanism forces attention scores to sum to one, causing the model to dump its residual attention onto early tokens even when they carry no relevant information. When using naive sliding window attention, removing these initial tokens causes the model's perplexity to spike and language generation to collapse completely. By explicitly preserving these initial sink tokens in the key-value cache, models can stream long conversations and generate fluent text indefinitely with constant memory usage.
Why this matters for your business
Leveraging attention sinks allows developers to deploy long-running chat applications that remain fluent over millions of tokens without expensive context-window expansions.
