TL;DR
An optimization technique that stores intermediate attention states during transformer-based autoregressive text generation to avoid redundant computation.
Transformer models generate text sequentially, which normally requires recalculating historical attention vectors at each new token generation step. By keeping the calculated key and value vectors in memory, a key-value cache allows the model to process only the newest token and retrieve the rest of the history. This mechanism drastically reduces computational overhead and inference latency during long-context conversations.
Why this matters for your business
It is the primary engine behind fast real-time chatbot responses and is crucial for scaling up simultaneous user requests and maximizing GPU efficiency.
