What MLPerf Endpoints Measures and Why It Matters to Buyers

What MLPerf Endpoints measures

MLPerf® Endpoints measures what inference systems deliver under production-like conditions. Rather than benchmarking a tuned configuration in isolation, it evaluates a served endpoint that combines the model, hardware, serving software, and networking under realistic load, reflecting how customers consume inference.

What buyers can learn

The full operating frontier, not a single point

An Endpoint result is a Pareto frontier covering operating points across concurrency levels. This shows throughput and responsiveness at loads corresponding to different workloads and service-level agreements (SLAs), from low-concurrency interactive use to high-concurrency serving.

Metrics tied to purchasing decisions

MLPerf Endpoints defines five primary metrics. Other measures are derived metrics:

• Concurrency: The number of simultaneous queries the system is serving, used as an approximate proxy for the number of users.
• Time to First Token P90: The time required for 90% of users to see the first response, measuring the delay before a response begins.
• System Throughput: Total system output, measured in tokens per second.
• System Throughput per Watt: A proxy for system total cost of ownership (TCO), using provisioned power to normalize throughput.
• Interactivity: Streaming speed per user, measured in tokens per second per user.

Results that follow the market

Endpoints accepts rolling submissions and publishes cohorts every few weeks. This allows results to cover current models and software rather than lagging market changes by months.

A peer-reviewed process

Each result has an explicit status:

• Verified: Passed MLCommons peer review.
• Provisional: Submitted, passed initial checks, and awaiting review.
• Unverified: Not submitted for review.

Buyers can weigh results according to their status. Each result also follows a defined audit process.

What Endpoints shows that other benchmarks do not

MLPerf Inference measures a system running a model in fixed scenarios, using a small number of predefined operating points for single-shot inference. MLPerf Endpoints measures the served endpoint as it is bought and consumed: under production-like load, across the full operating range, and for both single-shot and agentic or multi-turn tasks.

The whole curve, not fixed points

Endpoints publishes the complete Pareto frontier across concurrency. Two systems can achieve identical scores at a fixed point while behaving very differently at a buyer's actual operating point. The curve reveals that difference.

• Enterprise leaders: Evaluate the risk of purchasing excess capacity or selecting infrastructure that cannot support business demand. Applications with similar workload requirements can be mapped to different points on the tested system's curve.
• AI infrastructure decision-makers: Use measured load levels to assess capacity headroom and plan expansion based on throughput and responsiveness requirements rather than peak scores alone.
• Model decision-makers: Determine whether a model and serving configuration can meet expected demand while maintaining the required user experience. This can help narrow deployment options before integration effort and budget are committed, while model quality remains a separate selection criterion.

The system as sold, not only the configuration as tuned

Serving software, batching, scheduling, and networking are included in the Endpoints measurement because they are part of the product being purchased.

• Enterprise leaders: Request results for the configuration being purchased to reduce procurement and deployment risk and establish clearer expectations for supplier selection and acceptance testing.
• AI infrastructure decision-makers: Evaluate hardware and serving software together, directing investment toward configurations that provide useful capacity rather than assuming a faster accelerator will produce a better service.
• Model decision-makers: Assess how the selected model performs through the proposed serving environment and identify deployment options that require further validation before integration.

These evaluations can reduce avoidable rework and improve production planning, while application-specific quality, security, and integration checks remain necessary.

Making the throughput and experience tradeoff visible

Every deployment balances throughput against per-user experience. Endpoints makes this tradeoff explicit, while a fixed-scenario score cannot fully represent it.

• Enterprise leaders: Combine measured performance with actual pricing to assess the cost of delivering the responsiveness required by each application.
• AI infrastructure decision-makers: Identify operating points that maximize throughput while meeting latency and streaming-speed requirements, improving utilization without planning around peak capacity that fails those requirements.
• Model decision-makers: Consider performance limits alongside quality evaluations to determine which model and serving combinations are viable for each application. This supports decisions involving inference spending, service quality, and model choice, with experience targets determined by business needs.

Results published at market speed

Rolling submissions and publication cohorts every few weeks can provide results for models being evaluated today, rather than relying only on benchmark cycles measured in months.

• Enterprise leaders: Reduce the risk of making long-term purchasing decisions based on outdated performance assumptions and support investment cases with newer evidence as relevant results become available.
• AI infrastructure decision-makers: Use updated, comparable results to determine whether software improvements justify retesting existing infrastructure before purchasing additional capacity.
• Model decision-makers: Revisit deployment shortlists as supported models and serving stacks evolve, identifying options for further validation without assuming the newest release is automatically better.

Benchmark compatibility, system availability, and application-specific testing remain important parts of adoption, renewal, and upgrade decisions.

How to read Endpoints results

The recommended order is four steps:

1. Locate your workloads. Identify the workload type, such as an interactive assistant, high-volume batch process, or agentic pipeline. Each corresponds to a concurrency region on the published curve.
2. Set workload SLAs. Define the experience required by users, then select the necessary TTFT P90 and interactivity, measured in tokens per second per user. This removes operating points that do not meet the requirements, rather than removing entire systems.
3. Compare throughput at the selected point. Among systems that meet the SLAs at the relevant concurrency, compare per-system tokens per second. This represents the capacity being purchased.
4. Apply normalization. MLPerf Endpoints normalizes System Throughput by provisioned power, which is a proxy for cost. Buyers can also normalize by GPUs provisioned or dollars per GPU-hour. The price paid for a system or service may depend on purchasing volume, discounts, and other commercial factors. Provisioned power is the only normalization MLCommons officially guarantees. Other normalizations are derived and unverified.

In one sentence: Find the concurrency region matching the workload, filter for systems that meet the latency and interactivity requirements, then compare system throughput among the remaining results.

Why MLCommons is evolving toward MLPerf Endpoints

The move toward MLPerf Endpoints reflects how enterprise AI is bought, deployed, and consumed. Traditional inference benchmarks evaluated isolated hardware performance with static, single-shot models under controlled, predefined conditions. Modern AI buyers often acquire live, integrated serving stacks, APIs, and managed cloud endpoints intended for dynamic, multi-turn workloads such as AI agents.

As enterprise workloads move from simple queries to continuous production traffic, peak hardware output alone does not predict operational performance. MLPerf Endpoints measures the complete production system as sold, including the combined effects of batching, scheduling, networking, and serving software across a range of real-time load conditions.

This approach provides information for evaluating total cost of ownership, over-provisioning risk, and end-user experience under peak demand. It also makes benchmark results more directly relevant to procurement and deployment decisions.

Workloads, models, and datasets

Endpoints v1.0 includes six benchmarks. Each benchmark combines a task, a model, and a dataset. The task describes the work, the model performs it, and the dataset supplies the traffic used for measurement and the accuracy checks used to confirm that quality is maintained at the measured speed.

The six benchmarks include two single-shot summarization benchmarks, one single-shot reasoning benchmark, and three multi-turn agentic coding benchmarks.

Start with the workload rather than the leaderboard. Identify the task family that matches the application, then review the curves for the corresponding benchmarks.

Summarization, single-shot

Llama 3.1 8B Instruct on CNN/DailyMail is the small dense-model benchmark.

• It is relevant to high-volume, cost-sensitive text work, including summarization, extraction, and classification, as well as the serving of small fine-tuned models.
• It was carried forward from MLPerf Inference and therefore has the longest comparison history among these benchmarks.

GPT-OSS 120B on the MLPerf GPT-OSS set is a mid-size mixture-of-experts (MoE) model and was the most popular benchmark in recent MLPerf Inference rounds.

• It is suited to evaluating deployments of general-purpose open-weight models for chat, assistants, or document work.
• Its popularity provides a broad set of comparable submissions across vendors.

Reasoning, single-shot

DeepSeek-R1 on the reasoning set is a large MoE reasoning model that produces long thinking outputs before answering.

• Its curve is relevant to reasoning-heavy workloads such as complex analysis, mathematics, science, code generation, and planning steps within agent systems.
• Because outputs are long, sustained generation speed is important in addition to time to first token.

Agentic coding, multi-turn

Qwen 3.6, Kimi K3, and DeepSeek V4 Pro run on agentic tool-calling trajectories. The three models are measured on identical multi-turn agent traffic.

• Agentic inference involves turns that depend on earlier turns, contexts that grow as the agent works, and possible computation reuse across turns. These characteristics are not represented by single-shot tests.
• These curves are relevant when buying capacity for coding agents or other tool-calling pipelines.
• Since all three models run the same traffic, the results also support model comparisons on the same agent workload, not only system comparisons.

The four datasets

In Endpoints, the dataset has two functions. The performance set defines the traffic, including prompt lengths, output lengths, and turn structure, so the load reflects the behavior of the workload. The accuracy set verifies that the endpoint continues to produce correct answers at the measured speed.

CNN/DailyMail

Used for summarization with Llama 3.1 8B Instruct, CNN/DailyMail contains news articles paired with human-written summaries. It represents a standard summarization corpus and the traffic shape of document-processing pipelines, with moderate inputs and short outputs. The dataset downloads automatically with the benchmark.

MLPerf GPT-OSS set

Used for summarization with GPT-OSS 120B, this set uses mlperf_gpt_oss_performance to drive the load. The companion mlperf_gpt_oss_accuracy set checks quality against AIME25 competition mathematics, GPQA graduate-level science questions, and LiveCodeBench coding problems.

Reasoning set

Used with DeepSeek-R1, the reasoning set contains difficult prompts drawn from AIME, MATH500, GPQA, MMLU-Pro, and LiveCodeBench. These problems require long reasoning chains, stressing sustained token generation, which is important in reasoning and agent-planning workloads.

Agentic tool-calling trajectories

Used by all three agentic coding models, this is the single agentic dataset. It contains 613 multi-turn tool-calling trajectories hosted in MLCommons storage. The trajectories replay a coding agent's workflow through repeated model calls, tool results fed back into the model, and context that grows from turn to turn.

Accuracy is scored inline during the run, along with 200 tasks from SWE-bench Verified, in which the agent must resolve real GitHub issues end to end.

Conclusion

MLPerf Endpoints measures the system being purchased: the model, hardware, serving software, and networking together under production-like load. It provides the full Pareto frontier across concurrency, metrics connected to purchasing decisions, frequently updated results, and a defined peer-review process.

Buyers should begin with the workload, define the SLAs, and read the curve at the relevant operating point. The resulting throughput represents the capacity being evaluated and purchased.