Chapter 5: Techniques

5.5 Disaggregation

Separating prefill and decode onto independently scaling workers, and when that pays off.

Disaggregation combines three important ideas in inference engineering:

  1. Prefill is a compute-bound process that determines your TTFT, while decode is a memory-bound process that determines your TPS.
  2. Specialization improves performance in everything from kernel selection to inference engine parameter tuning.
  3. You can effectively parallelize model serving over multiple GPUs, or even multiple nodes, if you can avoid bottlenecks from lower-bandwidth interconnects.

When prefill and decode run on the same node under heavy traffic, they have a higher chance of interfering with one another. Ideally, prefill uses more compute resources, while decode uses more memory, and the two can co-exist efficiently. However, with larger batches and more compute-intensive optimizations, prefill and decode start competing for resources.

5.5.1 How Disaggregation Works

Disaggregation, or disaggregated serving, is the idea of separating prefill and decode into separate engines on separate GPUs or nodes.

Figure 5.17: Disaggregation assigns prefill workers to generate the first token and decode workers to generate subsequent tokens.
Figure 5.17: Disaggregation assigns prefill workers to generate the first token and decode workers to generate subsequent tokens.

Disaggregation turns LLM inference into a three-step process:

  1. The prefill engine takes the input sequence and generates a KV cache while computing the first token.
  2. The prefill engine sends the KV cache over the hardware interconnect to the decode engine.
  3. The decode engine computes all subsequent tokens.

In conditional disaggregation, the request is first sent to the decode engine, which checks if the input sequence is already cached or is short enough to handle locally:

  1. If it is, the decode engine handles prefill locally, skipping disaggregation.
  2. If it is not, the decode engine transfers the request to the prefill engine for disaggregated serving.

Conditional disaggregation is better for real-world traffic.

Another benefit of disaggregation is that with separate prefill and decode engines, you can optimize each engine individually and the system as a whole. For example, the compute-bound prefill engine requires a lower TP than the memory-bound decode engine.

5.5.2 When to Use Disaggregation

Disaggregation is very powerful but requires multiple GPUs and extra engineering work. You should reach for disaggregation only when:

  1. You are serving a large volume of traffic, starting at one hundred million to one billion tokens per day depending on model size.
  2. You are serving a larger model, at least a hundred billion parameters.
  3. Your traffic is prefill-heavy with long input sequences.

If either point one or two is not true, you’re likely wasting money on extra hardware for minimal performance gains. If point three is not true, you may be better off using the extra GPUs to scale replicas horizontally, as decode engines will be more efficient for short sequences or prefix cache hits.

A great use case for disaggregation is serving a frontier LLM in a code editor, where many developers are simultaneously passing in large and varied chunks of code as context. Tons of tokens, mostly prefill, on a trillion-parameter LLM is the textbook workload for disaggregation.

5.5.3 Dynamic Disaggregation with NVIDIA Dynamo

Dynamo provides production-ready support for disaggregation, with flexibility to handle heterogeneous real-world traffic.

Dynamo provides developer tools and pre-built optimizations to enable disaggregation:

  • A prefill queue to hold requests when all prefill engines are saturated.
  • Robust support for conditional disaggregation, with prefill routing based on configurable thresholds for ISL after prefix cache and prefill queue size.
  • Efficient NIXL-based KV transfer from prefill to decode engines with a kernel to transpose KV blocks between layouts when the engines have different TP configurations.

Combined, these features enable dynamic disaggregation, where the number of prefill and decode engines is configurable at runtime and can be adjusted over time to match the changing nature of incoming traffic.

Disaggregation does not need to be a one-to-one ratio between prefill and decode engines. While it’s simple to explain disaggregation in terms of a single prefill engine and a single decode engine, real systems have multiple of each.

The number of prefill and decode engines is written as xPyD, for example, 5P3D means five prefill and three decode engines working together to serve a single model deployment.

As systems grow more complicated, more potential bottlenecks appear. With disaggregation, the new bottleneck is prefill queue size. It’s important to not let the queue grow too large, both by setting a reasonable threshold for local prefill on the decode engine and by reconfiguring xPyD at runtime to allocate more resources to prefill if needed.

The other potential bottleneck in disaggregation is running out of KV cache on the decode engines under high load. Increase KV cache availability with quantization and KV cache offloading.