Understanding Key Metrics for Serverless LLM Inference

Introduction

When assessing serverless large language model (LLM) inference models and their providers, the focus often narrows down to one metric: median tokens per second. While this is an easy metric to publish and rank, it only provides a limited view of performance once a workload hits production. Different workloads encounter unique bottlenecks. For instance, a nightly summarization task benefits from sustained throughput, making median tokens per second a useful measure. However, a chat interface used by end-users requires quick and consistent initial token generation. For production services, factors like the slowest request times, error rates, and cost per response are critical, none of which are fully captured by median throughput alone. Optimizing the wrong metric can lead to a system that performs well on benchmarks but fails in real-world scenarios. This article delves into the metrics that truly matter for production serverless inference, clarifying what each measures and which workloads should prioritize them.

Key Takeaways

• No Single Fastest Provider: Across various models and providers, no single provider consistently emerges as the fastest. Performance can vary significantly depending on the model, with some models being served much faster by certain providers while others are slower.
• Availability Is Crucial: Availability is often overlooked in benchmarks but is a decisive factor. Some providers may offer fast models that are not reliably available, diminishing their value.
• First-Token Stability Over Speed: For most production applications, the consistency of the time to the first token is more important than sheer speed. Users notice the worst-case scenarios rather than average performance.
• Cost Efficiency: The cost per useful answer is crucial and is more influenced by model choice than by the provider's listed price. Selecting the right model for the task is a significant cost factor.

Throughput (Tokens per Second)

Throughput measures the rate at which a model generates tokens after starting, and it is often highlighted in benchmarks. This metric is suitable for workloads like batch jobs or offline processes where no user is waiting for a response. Throughput is typically measured as single-stream throughput, but in production, multiple requests occur simultaneously. Therefore, aggregate throughput under concurrency and how gracefully performance degrades under load are more relevant metrics. Throughput also depends on model architecture, as mixture-of-experts models can generate tokens faster than similarly sized dense models.

Time to First Token and Its Stability

For interactive applications, time to first token (TTFT) is the critical metric that defines user experience. In a chat interface, TTFT is the delay between initiating a request and seeing the first response. Consistency in TTFT is crucial; a model that delivers the first token quickly and predictably can feel instantaneous, even if its overall throughput is lower. It’s essential to measure TTFT as a range, comparing the median to the 95th percentile, to capture the experience that the median alone might hide.

Tail Latency (p95 / p99)

Tail latency measures the end-to-end time of the slowest requests, represented by the 95th and 99th percentiles. It is essential for service-level objectives, HTTP timeouts, and capacity planning. At production traffic levels, tail latency is not an outlier but a consistent aspect of the workload. A large gap between median and tail latency can indicate server path issues.

Reliability and Availability

Speed is irrelevant if a request fails or the model is unavailable. Availability refers to whether a model can be accessed without dedicated infrastructure, while reliability concerns whether requests succeed once the model is available. Testing specific models intended for deployment is important, as newer models may have reliability issues.

Cost per Useful Result

The true cost metric should consider the cost of a completed, useful response, not just the listed price per million tokens. Model choice and routing capabilities significantly impact this cost, especially with reasoning models that incur additional thinking tokens. Effective cost management involves selecting the right model for each task and using an inference router to manage this routing automatically.

Cold Starts and Burst Behavior

Serverless systems introduce cold starts, where the first request after an idle period incurs a startup delay. This metric is crucial for bursty traffic patterns. Platforms offering keep-warm or provisioned capacity should be evaluated for their ability to handle bursts efficiently.

Output Fidelity

Output fidelity assesses whether a model's response is correct, complete, and of the expected quality, beyond simply returning HTTP 200. This includes checking for silent truncation or quality changes due to quantization. Ensuring valid output involves budgeting enough tokens for reasoning models and verifying the precision level served.

Operational Fit

Operational fit is about how well a platform aligns with development practices. While many providers offer OpenAI-compatible APIs, compatibility extends beyond endpoint shapes to include parameter handling, accurate token usage reporting, streaming reliability, and data residency options. Ensuring the platform supports your application's specific needs is critical.

Choosing the Metrics for Your Workload

Metrics should be selected based on the specific requirements of an application. Each workload type has primary and secondary metrics that are most relevant:

• Interactive Chat / Streaming UI: Prioritize TTFT stability, reliability, and tail latency.
• Batch / Offline Generation: Focus on sustained throughput under concurrency and cost per result.
• Retrieval-Augmented Generation / Summarization: Emphasize TTFT, cost per result, and reliability.
• Production Service at Scale: Reliability, availability, and tail latency are key, with median throughput as a secondary concern.

FAQ

What is the most important metric for serverless inference?

There is no universally important metric; it depends on the workload. Interactive applications value TTFT stability and reliability, while batch processes focus on sustained throughput and cost per result. Median tokens per second is a starting point but not decisive for most deployments.

Why look at p95 latency instead of the median?

The median represents typical requests, but p95 highlights the experience of the unluckiest users, which can be significant at high traffic volumes. A provider can excel in median latency but fail at p95.

Do reasoning models cost more on serverless inference?

Yes, they are significantly more expensive than their list price suggests. Reasoning models generate thinking tokens billed as output, leading to costs that are much higher per request compared to smaller models.

Why do benchmark results differ between providers for the same model?

Differences arise from hardware, batching strategies, quantization levels, and pool provisioning. Performance can vary widely between providers depending on the model tested.

How many trials are needed for a trustworthy benchmark?

At least 25 trials per model and scenario are recommended, with a few warmup requests discarded. This sample size allows for stable median and indicative p95 reporting.

Conclusion

Serverless inference performance isn't defined by a single metric. The commonly used median tokens per second metric only addresses batch throughput. Key metrics for production deployments include model availability, first-token latency, tail latency, and the actual cost per completed answer. Before choosing a provider, it's crucial to test these metrics with your own workload to understand the underlying infrastructure better.