TL;DR
An AI-serving architecture that separates the compute-heavy input prefill phase and the memory-bound output decode phase onto independent pools of hardware.
Traditional large language model serving forces individual processors to alternate between processing massive input prompts and sequentially generating output tokens. Disaggregated inference optimizes this process by routing input processing to high-compute prefill hardware, transferring the resulting key-value cache across the network to memory-bandwidth-optimized hardware that generates response tokens. This partition ensures both phases run under ideal hardware configurations, maximizing resource utilization while minimizing end-to-end latency.
Why this matters for your business
This architecture allows enterprises to handle unpredictable, high-volume workloads from agentic systems efficiently without suffering from massive hardware bottlenecks.
