MLCommons Reports Record Participation in MLPerf Inference v6.1 Results
MLPerf Inference v6.1 sets a new participation record MLCommons has published results from MLPerf Inference v6.1, its industry standard benchmark suite for AI inference. The lat...
By AI Engineering Team
MLPerf Inference v6.1 sets a new participation record
MLCommons has published results from MLPerf Inference v6.1, its industry-standard benchmark suite for AI inference. The latest release received submissions from 30 organizations, the highest number in the benchmark's history, and adds two tests designed around newer AI deployment patterns.
The results also include the first peer-reviewed performance data for several recently released or upcoming AI platforms. In some workloads, performance improved by as much as 5.7x compared with results from one year earlier.
The open-source MLPerf Inference suite measures system performance in an architecture-neutral, representative, and reproducible way. Its published results are intended to provide empirical data for organizations evaluating and deploying AI systems, while supporting comparisons across hardware, software, and deployment configurations.
New tests address multi-step and agentic inference
MLPerf Inference v6.1 introduces two tests that reflect the industry's movement toward more complex, multi-step inference workloads in datacenters and at the edge.
End-to-End Retrieval-Augmented Generation
The End-to-End Retrieval-Augmented Generation (RAG) benchmark evaluates a question-answering pipeline built from several AI models and components. The process includes:
- An embedding model converts a query into a vector.
- A retriever obtains candidate passages from a vector database.
- A re-ranker refines the candidate list.
- One or more large language models reason over the retrieved information and produce an answer.
The benchmark measures two related tasks: ingesting a document corpus to create a vector database, and answering queries against an existing vector database.
Edge Agentic Inference
The Edge Agentic Inference benchmark measures multi-turn workloads, including agentic coding, rather than single-shot interactions. These workloads maintain an expanding conversation history, with each query depending on earlier exchanges. They also combine evidence gathering and reasoning, repeating the process as needed before producing a result.
This approach can support more complex uses than single-shot inference, but it also increases demands on AI computing resources. Edge deployments introduce additional constraints because they may serve one user at a time with fixed memory, processing capacity, power, and context limits.
The test uses the framework and methodology of the forthcoming MLPerf Agentic datacenter performance benchmark, adapted for an edge model and quantization. It includes a single-stream coding workload, latency metrics, and a statistically robust accuracy gate. The benchmark measures both workload accuracy under time constraints and performance for a deterministic workload whose accuracy checks are built into the workload.
Support for speculative decoding
MLPerf Inference 6.1 also adds support for speculative decoding, an optimization that predicts and verifies multiple tokens during a single forward pass. The technique is now supported in the interactive scenario for two inference benchmarks, with additional support for the GPT-OSS task.
Miro Hodak, MLPerf Inference working group co-chair, said the new tests are intended to reflect multi-step, multi-component pipelines and agentic systems being deployed in datacenters and on edge devices. He also said the benchmark would continue to include performance optimizations used in production inference systems.
New hardware and larger systems
The submissions cover hardware across accelerators, boards, workstations, and servers. Five new processors or accelerators appear in the results:
- AMD Ryzen AI Max+ 395
- AMD Instinct MI350P
- Intel Arc Pro B70
- NVIDIA Rubin, listed in preview
- NVIDIA Vera Rubin NVL72, listed in preview
The release includes the largest system submitted to MLPerf Inference so far, with 512 accelerators. It also includes two heterogeneous systems. One combines high-performance networking with accelerators from two vendors, while the other is geographically distributed across the Pacific Ocean.
Performance gains across workloads
Performance continued to improve across several tests. In the Visual Language Model (VLM) test, the best per-accelerator server result in this round was 2.99x higher than the corresponding result in v6.0, released six months earlier.
For the Deepseek R1 test, the best per-accelerator server result was 5.7x higher than the result in v5.1 from one year earlier. These gains can allow systems to serve more users or support more demanding inference workloads.
Frank Han, MLPerf Inference working group co-chair, said the results show the effects of improvements across hardware, software, and system architectures. He added that the benchmark data can help customers evaluate cost and performance when procuring and deploying AI systems in datacenters and at the edge.
Thirty organizations submit results
MLPerf Inference 6.1 received submissions from these 30 participating organizations:
- AMD
- ASUSTeK
- Atlas Inference
- Cisco
- CoreWeave
- Crusoe
- Dell
- Fujitsu Limited
- GigaComputing
- Hewlett Packard Enterprise
- Intel
- Inventec Corporation
- KRAI
- Lambda
- MangoBoost
- Microsoft Azure
- MiTAC
- Nebius
- NVIDIA
- Oracle
- Orrick
- Quanta Cloud Technology
- RedHat
- ScitiX
- Supermicro
- Telecommunications Technology Association
- VibeHPC
- Wiwynn
- Individual contributor Naeem Khoshnevis
Atlas Inference, Crusoe, Orrick Industries LLC, ScitiX, VibeHPC, and individual contributor Naeem Khoshnevis were first-time submitters.
More than half of the submitters used MLCommons' new API-centric harness. The harness is designed as the foundation for the next-generation MLPerf Endpoints benchmark suite. It uses a client-server architecture and industry-standard APIs to send inference queries and return results, more closely representing datacenter deployments.
MLCommons says MLPerf Endpoints will eventually replace Inference for datacenter benchmarking. The adoption of the API-centric harness is intended to support that transition.
Accessing the results
MLPerf Inference v6.1 results are available on the MLCommons benchmark pages for:
Supplemental statements from submitters provide additional context about individual submissions and results.
About MLCommons
MLCommons is an open engineering consortium with more than 130 members and affiliates. Its work brings together academia, industry, and civil society to measure and improve AI systems.
The organization began with the MLPerf benchmarks in 2018. Those benchmarks developed into industry metrics for measuring machine learning performance and improving transparency around machine learning techniques.