Understanding the Impact of Continuous Batching on Latency Performance

Continuous batching is a fundamental component of modern large language model (LLM) serving systems. It is a standard feature across all major engines and is frequently highlighted in product comparison charts for its significant impact on throughput. Benchmarks have shown throughput improvements of up to 23x over basic setups, with some earlier systems achieving as much as 36.9x. These enhancements primarily concern throughput and median (p50) latency, as demonstrated in various studies, including Anyscale's benchmark. However, the effect on users experiencing the longer end of latency (p99) is less understood. This article explores how continuous batching can alter the entire latency profile, not just the average.

Latency Dynamics

Continuous batching shifts latency variance rather than simply increasing it, challenging the notion that it necessarily worsens tail latency (p99). In static batching, delay primarily occurs at admission, which can create bottlenecks under heavy loads. Continuous batching allows for nearly instantaneous admission but can introduce occasional pauses, or jitter, during token streaming. The specific latency patterns and trade-offs depend on workload and engine configuration.

Test Setup

The tests discussed here were conducted using a DigitalOcean H200 GPU Droplet running vLLM v0.24.0 with Llama 3.1 8B, utilizing a mixed-length traffic trace. This setup included:

• Continuous vs. static-style (gated, B=16) admission on the same server with identical conditions.
• Arrival rates increased from 1 to 20 requests per second for the continuous arm.
• Chunked prefill toggled on and off to observe its effects.

Key Findings

1. The well-known 23x throughput improvement is related to p50 latency, not p99.
2. Continuous batching's tail latency issues arise from two separate mechanisms: long prefills stalling in-flight decodes and preemption under KV cache pressure.
3. Modern vLLM defaults mitigate the first mechanism by enabling chunked prefill with a decode-first scheduling policy.
4. The second mechanism, preemption, remains a challenge, as running requests may be evicted under KV cache pressure, necessitating re-computation.

Experiment Results

The experiments demonstrated that:

• Gated/static admission significantly increases time-to-first-token (TTFT), with median TTFT being much higher compared to continuous batching.
• Continuous batching at higher request rates results in increased inter-token gaps, leading to a rougher token stream experience.
• Disabling chunked prefill exacerbates these gaps, confirming the importance of this mitigation strategy.

Conclusion

Continuous batching significantly improves median latency and throughput in LLM serving systems. However, it can also lead to increased p99 latency under certain conditions. Proper configuration, such as enabling chunked prefill, can mitigate some of these issues. Understanding and measuring these dynamics allows for informed decisions about batching policies based on specific workload requirements.

References

For further reading, various academic papers and documentation provide in-depth insights into the mechanisms and optimizations discussed here.