Understanding Latency Metrics: Why Median Benchmarks Can Be Misleading for AI Workloads

Most published benchmarks for AI inference highlight a single figure, such as the median time to first token or peak tokens per second. Although these figures aren't inaccurate, they often don't represent real-world conditions, such as high concurrency or cold starts. This discrepancy can lead to unexpected latency in production environments.

This guide presents two main arguments regarding latency metrics in AI workloads. Firstly, the shape of the latency distribution (p95 and p99) is more relevant than the median for understanding user experience. While the median reflects a typical request, it often misses the slowest 1% of requests, which are the ones users notice and complain about.

Secondly, when dealing with AI agents making sequential model calls, focusing on per-call latency can be misleading. Task completion time, which is the aggregate time across all calls, is the correct metric to consider. The likelihood of encountering a tail event increases with each additional call in a sequence.

This article will cover how to interpret latency percentiles, break down latency into meaningful metrics, explore the impact of agent chains on analysis, provide a checklist for evaluating benchmarks critically, and share a test protocol for real-world traffic evaluation.

Understanding Latency Terms

If you're new to latency analysis, here's a quick glossary:

Percentile: The value under which a given percentage of requests complete. For example, p99 means 99% of requests are faster than this value.
Median (p50): The middle value, with half of requests being faster and half slower.
p95 / p99: These percentiles show the slowest 5% and 1% of requests, respectively.
Time to First Token (TTFT): The time from sending a request to receiving the first output token.
Inter-token Latency: The delay between receiving subsequent tokens.
Total Completion Time: The entire duration for a request from start to finish.
Task Completion Time: The sum of completion times for all calls in an agent's task.
Concurrency: The number of requests processed simultaneously.
Cold Start: Additional delay when a model needs to load before processing requests.
Throughput: Work done per unit time, often measured in tokens per second.

Why Percentiles Matter More Than Medians

A median TTFT might be 300 milliseconds, but if the p99 is 4 seconds and you handle 10,000 requests daily, 100 requests will experience delays significantly longer than the median. This isn't noise; it's an inherent part of the distribution. For a single request, understanding the entire latency distribution is crucial. For sessions with multiple requests, this is even more critical, as the likelihood of encountering slow requests compounds.

Analyzing Percentile Divergence in AI Inference

Several factors contribute to the divergence between p50 and p99 in inference workloads:

Queueing: Requests might wait for others to process first.
Batching: Requests may wait for a batch to fill before processing starts.
Cold Starts: Increase TTFT significantly after idle periods.

These factors illustrate why real-world performance can differ from benchmark results.

When Median is Relevant

For high-volume, latency-insensitive processes like batch workloads or content moderation, throughput and cost per token are more significant than individual request timings.

Choosing the Right Latency Metric

Latency can be decomposed into three primary metrics, each suited to different workloads:

Time to First Token: Crucial for chat interfaces where users expect immediate feedback.
Inter-token Latency: Important for smooth, continuous outputs like voice responses.
Total Completion Time: Essential for tasks where no one is actively monitoring the output stream, like batch jobs.

Task Completion Time in Agent Workloads

In agent workflows, where tasks often involve multiple sequential model calls, the task completion time (the cumulative latency across all calls) is a more relevant metric than per-call latency. Misunderstanding this can lead to underestimating the real impact on user experience.

Correcting Misconceptions About Tail Probability

It's a common misconception that a 10-call sequence with a 1% tail rate per call almost guarantees a tail event. In reality, the probability is around 9.6%, which is significant but not near-certainty.

Measured Results: Benchmark Winners Aren't Always Task Winners

A real-world test showed that a model with the fastest median TTFT didn't complete tasks the quickest due to slower generation speeds during sequential calls. This demonstrates the importance of evaluating the right metrics for your specific workload.

Critical Evaluation of Latency Benchmarks

When reading latency benchmarks, consider:

Reported Percentile: Does the benchmark include only the median?
Concurrency Level: Was the test conducted at a realistic concurrency?
Prompt and Output Length: How do these affect the reported latency?
Instance State: Were measurements taken with a warm or cold instance?
Measurement Window: Does the time of day affect results?
Streaming Enabled: Is TTFT measured with streaming?

Decision Framework for Latency Evaluation

Interactive Workloads: Focus on p95 and p99.
Agent Pipelines: Assess task completion time.
Batch Workloads: Consider throughput and cost, not latency tails.

Conclusion

Latency benchmarks often highlight flattering metrics like medians, which can be misleading. It's essential to consider the full latency distribution for single requests and the aggregate task time for agent workloads. Use the test protocol provided to evaluate your own system's performance.

Understanding Latency Metrics: Why Median Benchmarks Can Be Misleading for AI Workloads