Understanding Long-Context LLM Serving: Balancing Memory, Latency, Cost, and Accuracy

What Long Context Means and How It's Different from Serving

In the realm of large language models (LLMs), "context" encompasses everything sent in a single request—prompts, documents, code, and conversation history. It is quantified in tokens, where roughly 1,000 tokens equate to 750 words. A typical 128K-token window can hold text equivalent to a novel, and it's a standard today. For instance, models like the Llama 3.1 and GPT-OSS-120B support such windows. More advanced models, like Claude Sonnet 4.6, can handle up to 1 million tokens, which is about ten novels.

The specifications indicate the model's capacity, but they don't reflect real-world performance when numerous users send large inputs concurrently. Model capability is the support aspect, while serving pertains to performance, with significant differences arising in four key areas, which are explored below.

Prefill and Decode

Inference in LLMs involves two stages. Prefill is where the model processes the entire input simultaneously before generating any output, being limited by GPU computational speed. Decode, on the other hand, generates responses one token at a time, constrained by how quickly data moves from GPU memory. Long inputs primarily affect prefill, while long outputs impact decode.

KV Cache

For each token processed, models generate and store two vectors (key and value) to avoid rereading the entire input for every new token. This forms the KV cache, which resides in GPU memory and scales with input size.

Tradeoff 1: Memory

The memory footprint of the KV cache can be calculated with:

2 × layers × KV heads × head dimension × sequence length × bytes per element

This formula accounts for two vectors (key and value) per token. For instance, using Llama 3 70B in BF16 precision (80 layers, 8 KV heads, head dimension 128) with a 128K-token input, the cache requires approximately 43 GB of GPU memory. Given that GPUs like the NVIDIA A100 or H100 have 80 GB total, a large context can quickly exceed available memory, necessitating distribution across multiple GPUs, which complicates serving and increases latency.

Batch processing is hindered as well. Long-context requests limit the ability to batch multiple requests, leading to inefficient GPU memory usage. Techniques like PagedAttention can reclaim some memory, but still require significant resources as context lengths increase.

Tradeoff 2: Latency

Latency refers to the time taken between sending a request and receiving a response. Two components affect user perception: time to first token (TTFT) and the speed of the answer stream. TTFT is particularly inflated by long contexts due to the quadratic increase in attention computation. Optimizations like FlashAttention can reduce some inefficiencies, but the fundamental quadratic growth remains unavoidable.

Additionally, long requests can delay shorter ones queued behind them, affecting overall system responsiveness and damaging the user experience in interactive applications.

Tradeoff 3: Throughput and Bandwidth

Even with sufficient computational power, decode processes face limitations due to memory bandwidth. Each output token generation involves reading the entire KV cache from high-bandwidth memory. As cache sizes increase, throughput is constrained by memory bus speed, not computational capability, leading to reduced tokens-per-second rates.

The cost of processing long contexts typically exceeds proportional token costs due to inefficiencies and resource saturation, reflected in tiered pricing models.

Tradeoff 4: Accuracy

Long inputs challenge model accuracy, often resulting in degraded performance compared to shorter contexts. Factors like effective context length and the position of relevant information within the input can impact accuracy significantly. Increasing latency due to retries for incorrect answers further compounds the issue, as demonstrated by time-to-correct-answer (TTCA) metrics.

Accuracy-aware routing strategies can mitigate some latency by optimizing model selection based on input characteristics, helping reduce TTCA across different models.

Relevance of Retrieval-Augmented Generation (RAG)

Despite the availability of large context windows, retrieval remains crucial for efficient LLM operation. Utilizing a vector database to retrieve only relevant slices of information reduces computational and memory demands, enhances model reliability, and maintains responsiveness.

Skipping retrieval leads to increased costs, quality degradation, and potential request failures due to memory constraints, while retrieval infrastructure, though costly upfront, offers long-term operational savings.

Practical Considerations

Systems leveraging both long contexts and retrieval can optimize performance and costs. Long contexts with caching are beneficial for stable, reusable datasets, while RAG excels with large or dynamic datasets. Precision tasks also favor RAG for maintaining accuracy.

Ultimately, choosing between these approaches depends on the specific use case, traffic patterns, and system architecture, with many systems adopting a hybrid model for optimal efficiency.