Disaggregating LLM Inference: The Shift to Separate Hardware for Prefill and Decode

*Exploring the division of prefill and decode processes in language model inference across different hardware, its implementation in production, and when it's worthwhile.*

### Introduction

Consider a bustling restaurant kitchen on a busy evening. One chef is overwhelmed with both chopping vegetables and finishing dishes, leading to delays. In a well-organized kitchen, prep cooks handle chopping, while line cooks focus on plating. Similarly, in large language models (LLMs) inference, a single GPU often handles both prefill (processing prompts) and decode (generating responses) tasks, causing delays. Leading systems now separate these tasks onto different hardware, enhancing efficiency.

![Illustration for: Consider a bustling restaurant...](https://storage.googleapis.com/xfinit-blogs-scraper-assets-664708921442/blog-assets/images/c83da588-4c91-4ee0-8668-5cf6949cce90.jpg)

### What is Prefill/Decode Disaggregation?

**Prefill/decode disaggregation** involves using separate GPU pools for each phase of LLM inference. Prefill, which involves processing the input, runs on one set of GPUs, while decode, which generates the response token by token, runs on another. Once prefill is complete, the model’s memory (KV cache) is transferred to a decode GPU to generate the response.

### Why Separate the Phases?

The two phases of LLM inference demand different GPU resources. Prefill is compute-intensive, while decode requires significant memory bandwidth. Separating them reduces interference and delays caused by long prompts in shared environments. However, this requires additional hardware, network transfers, and operational complexity.

### Competing GPU Resources

Every LLM request involves a prefill phase, which processes the entire prompt, and a decode phase, which generates the response. The prefill phase is compute-heavy, while decode is memory bandwidth-intensive. Traditionally, both phases compete for the same resources on a single GPU, leading to inefficiencies.

### Solving the Issue: Disaggregation

Disaggregation addresses these inefficiencies by using separate GPU pools for prefill and decode tasks:

- **Prefill Pool:** Handles incoming prompts.
- **Decode Pool:** Focuses on generating tokens.

This separation alleviates bottlenecks, allowing each pool to be optimized for its specific task, thereby enhancing performance and reducing latency.

### Implementations in Production

Several systems have adopted this approach:

- **DistServe:** Demonstrates the advantage of separating prefill and decode for optimizing latency targets.
- **Mooncake:** Uses a KV cache-centric design to handle traffic efficiently at scale.
- **DeepSeek:** Implements separate clusters for prefill and decode, optimizing cost and efficiency.
- **NVIDIA Dynamo:** Offers a framework for disaggregated serving, with significant throughput gains reported.

### Challenges and Considerations

Disaggregation increases hardware requirements and system complexity. It’s beneficial primarily for large-scale deployments with constant high traffic and latency-sensitive applications. For smaller applications, traditional unified systems might be more cost-effective and simpler to manage.

### Conclusion

Disaggregated inference is a promising direction for large-scale AI applications, offering improved efficiency and reduced latency. However, it’s not a one-size-fits-all solution and should be considered based on specific workload demands.

### Hardware Considerations

On certain infrastructures, disaggregated setups can be mapped to specific GPU configurations, optimizing compute and memory bandwidth needs for prefill and decode tasks, respectively.

### When to Consider Disaggregation

This approach is most beneficial when:

- **Traffic Volume:** GPUs are constantly busy.
- **Latency Consistency:** Interactive applications require consistent response times.
- **Prompt Length:** Applications handle long or complex prompts.

For many deployments, simpler optimizations may suffice, and disaggregation should be reserved for when those optimizations are no longer effective.

### Conclusion

Disaggregated inference is one of many techniques to optimize LLM serving. It should be viewed as part of a broader strategy, particularly useful for large, latency-sensitive workloads.