Where the Industry Is Investing: MLPerf Inference v6.1

A broad field of submitters and new benchmark directions

MLPerf Inference v6.1 provides a broad view of current industry engineering priorities. The round includes a record number of submitters, newly introduced hardware, and a shift toward agentic and end-to-end inference benchmarking.

Highlights include 30 participating organizations, new accelerators that improve per-device performance, and the introduction of a multi-turn agentic benchmark.

Submitters and submissions

The round included 30 submitters from silicon vendors, system builders, cloud and neocloud providers, and specialized inference-software companies. Several entries were joint submissions involving two organizations: Dell_AMD, Dell_MangoBoost, RedHat_Intel, and RedHat_Supermicro.

Altogether, 120 systems were submitted across the Datacenter and Edge suites and the Closed and Open divisions.

Benchmark coverage

MLPerf Inference divides its benchmarks into Datacenter and Edge categories. Each benchmark can support one or more scenarios: Offline, Server, Interactive, SingleStream, or MultiStream.

Submissions are also classified as Closed or Open. Closed submissions must use a model mathematically equivalent to the reference implementation, keeping the model fixed for direct comparisons.

MLPerf Inference v6.1 includes 10 Datacenter benchmarks and 6 Edge benchmarks. Two are new: End-to-End RAG and Agentic Edge Inference. Two existing benchmarks were updated. VLM, based on Qwen3, now includes an Interactive scenario, while the GPT-OSS-120B Interactive scenario allows speculative decoding.

Datacenter benchmarks

• GPT-OSS-120B, updated: 54 systems, Server, Offline, Interactive
• Llama2-70B-99.987: 39 systems, Server, Offline, Interactive
• Llama3.1-8B: 31 systems, Server, Offline, Interactive
• DeepSeek-R1: 32 systems, Server, Offline, Interactive
• VLM, updated: 19 systems, Server, Offline, Interactive
• Text-to-video: 15 systems, SingleStream, Offline
• Whisper: 23 systems, Offline
• DLRM-v3: 3 systems, Server, Offline
• RGAT: 5 systems, Offline
• End-to-end QA, new: 1 system, Offline
• End-to-end vector-DB, new: 1 system, Offline

Edge benchmarks

• Agentic Edge, new: 5 systems, SingleStream
• YOLO-95: 3 systems, SingleStream, MultiStream, Offline
• Whisper: 2 systems, Offline
• YOLO-99: 1 system, SingleStream, MultiStream, Offline
• ResNet: 1 system, SingleStream, MultiStream
• Llama3.1-8B-Edge: 1 system, SingleStream, Offline

The most popular Datacenter benchmark was GPT-OSS-120B, followed by Llama2-70B. This was the first round in which GPT-OSS-120B led participation, after Llama2-70B had topped the list in previous rounds. The result reflects increased participation in Mixture-of-Experts models.

Performance compared with v6.0

The best per-accelerator Offline and Server scores were compared with results from MLPerf Inference v6.0.

The percentages are reported in the results analysis. Scores are scaled independently for each model, so comparisons are meaningful only within a row. RGAT's best Offline score was marginally below its v6.0 result.

The largest gains occurred in VLM and DeepSeek-R1 for both Offline and Server scenarios. These benchmarks were submitted on a new Preview-category system powered by NVIDIA Vera Rubin. The best scores for the other benchmarks used the same hardware as in v6.0, so their gains primarily reflect software-stack and algorithmic improvements.

Long-term Llama2-70B gains

Llama2-70B has been part of MLPerf Inference since round v4.0 in early 2024, making it a useful benchmark for tracking performance over time. Median per-accelerator performance for Server submissions has improved 5.58 times across six rounds.

Three factors account for much of the improvement:

1. Reduced numerical precision: Some v6.1 submissions use FP4, while earlier rounds used FP8. MLPerf Inference accuracy requirements mean these gains did not come with a loss of accuracy.
2. New accelerator generations: Older accelerators used in initial Llama2-70B rounds have been replaced by newer, higher-performing devices.
3. Software improvements: Performance has continued to improve even on the same hardware through advances in software and inference optimization.

DeepSeek-R1 gains

DeepSeek-R1 shows a similar pattern over a shorter period. Within one year, best per-accelerator performance increased 2.7 times in Offline and 5.7 times in Server scenarios. The Interactive scenario, which had only two submission rounds, improved 2.7 times during the same period.

Results highlights

More multi-node systems

Multi-node inference has continued to grow. This round set a record with 16 multi-node submissions, continuing an upward trend that began in round 4.1 two years earlier.

The scale of submitted systems also increased. The previous round's record was 288 accelerators. Crusoe submissions 6.1-0026 and 6.1-0027 raised that figure to 512 accelerators. Submission 6.1-0027 also set a new MLPerf Inference record for generated tokens per second, producing almost 5.8 million tokens per second in the GPT-OSS-120B Offline test.

Hybrid accelerator deployments

MLPerf Inference v6.1 included two hybrid submissions using different accelerator types in the same system. These deployments require submitters to account for differences in the devices' computational capabilities.

Cisco submitted the first cross-vendor heterogeneous accelerator deployment, combining eight NVIDIA H200 GPUs and eight AMD Instinct MI350X GPUs in one inference pool connected by a Cisco G200 Network.

New accelerators and systems

The round introduced a diverse set of accelerators and systems, ranging from on-device components to a next-generation rack-scale platform. The new submissions covered multiple accelerator vendors and deployment scales.

Submitter highlights

Participating organizations supplied supplemental descriptions of their v6.1 submissions. Their reported highlights included the following:

• AMD: Expanded from three models in v6.0 to seven models across the AMD Instinct MI355X, MI350X, and MI350P platforms.
• ASUSTeK: Reported 113,455 tokens/s in the Llama2-70B Server scenario.
• Atlas Inference: Submitted an agentic system intended to perform inference without relying on a datacenter round trip.
• Cisco: Combined eight NVIDIA H200 and eight AMD Instinct MI350X GPUs in one inference pool connected by a Cisco G200 Network.
• CoreWeave: Reported 1.16 million tokens/s aggregate throughput on GB300 NVL72, with per-GPU Offline throughput improving 17% since MLPerf v6.0.
• Crusoe: Reported more than 5.5 million tokens/s on GPT-OSS-120B and 2.9 million tokens/s on DeepSeek-R1 using AMD MI355X, along with NVIDIA GB200 results from Crusoe Managed Kubernetes and Managed Slurm.
• Dell: Reported performance from a PowerEdge XE9785L with eight AMD Instinct MI355X GPUs across Llama2-70B and GPT-OSS-120B.
• Fujitsu: Reported an improvement of approximately 50% over its previous-round results.
• GigaComputing: Tested a B300 GPU paired with a 120-core Intel Xeon 6979P.
• Google: Focused its MLPerf 6.1 submission on DeepSeek-R1 as participation expanded toward large Mixture-of-Experts architectures.
• HPE: Reported 136,000 tokens/s from two HPE Compute XD690 servers with NVIDIA Blackwell Ultra GPUs, including 8,500 tokens/s per GPU on DeepSeek-R1.
• Intel: Reported that, on the same silicon and socket count as v6.0, Llama3.1-8B Server throughput on Xeon 6980P increased 2.4 times, or 142%, while Offline throughput increased 56% through software changes.
• Inventec: Submitted platforms supporting liquid- and air-cooling configurations.
• Krai: Submitted the first result from KRAI/Gentic, an automated agentic full-stack optimization framework designed to search for faster kernels and integrate them with vLLM.
• Lambda: Reported results for Qwen3-VL-235B-A22B-Instruct and GPT-OSS-120B on NVIDIA B200 and GB300 chips. GPT-OSS-120B throughput increased by up to 8.85% over the same workload in MLPerf Inference v6.0.
• MangoBoost: Submitted prefill/decode-disaggregated results on AMD Instinct GPUs and a multi-region cluster spanning four sites on two continents, operating as one endpoint at 97% scaling efficiency.
• MiTAC: Submitted results from the G8825Z5, G4826Z5, and G4520G6 AI/HPC server series across generative AI and large language model workloads.
• Nebius: Was one of two submitters using next-generation silicon.
• NVIDIA: Reported that the NVIDIA Vera Rubin NVL72 platform achieved up to 2.5 times higher throughput on DeepSeek-R1 and up to 3.7 times higher throughput on Qwen3-VL than the prior generation.
• Oracle: Submitted results using NVIDIA GB300 and B300, AMD Instinct MI355X, and Intel X12 platforms.
• Orrick Industries: Reported Llama-3.1-8B throughput of 274,927 output tokens per second using published model weights in NVFP4 precision with an FP8 KV cache.
• Quanta Cloud Technology: Submitted two AI platforms addressing different deployment, performance, and scaling requirements.
• Red Hat: Demonstrated inference on Grace-Blackwell systems orchestrated by OpenShift and on general-purpose Xeon CPUs, using the same open-source vLLM serving stack.
• SciTiX: Reported 60,411 tokens/s in the DeepSeek-R1 Offline scenario and 59,668 tokens/s in Server, under the required accuracy and latency constraints.
• Supermicro: Submitted results for NVIDIA B300, AMD Instinct MI355X, Intel Arc Pro B70, and Intel Xeon 6/6+ CPUs.
• TTA: Evaluated the DeepGadget dg5W for enterprise-grade LLM serving without massive datacenter infrastructure.
• VibeHPC: Reported 53,267.2 samples/s on Whisper Offline using an eight-GPU B300 system. The same system achieved 156,867 tokens/s in Llama 3.1-8B Server and 161,767 tokens/s in Offline.
• Wiwynn: Reported 63,991 tokens/s in the Offline scenario, with p99 TTFT of 336 ms against a 3,000 ms limit and accuracy above 83%.
• Naeem Khoshnevis: Reported nearly 7,800 tokens/s on a single NVIDIA H200 using FP8 quantization and an optimized inference stack for Llama 3.1-8B.

Summary

MLPerf Inference v6.1 was one of the benchmark's largest rounds. Thirty organizations submitted 120 systems across the Datacenter and Edge suites and the Closed and Open divisions.

The submissions ranged from single-accelerator edge devices to rack-scale platforms with hundreds of accelerators, all measured under a common methodology. Several trends stood out:

1. Mixture-of-Experts workloads gained prominence. GPT-OSS-120B became the most popular model for the first time, while DeepSeek-R1 and Qwen3-VL also attracted substantial participation.
2. Agentic and end-to-end benchmarking expanded. The suite moved beyond single-shot evaluation with new agentic and end-to-end workloads.
3. New hardware appeared across the deployment spectrum. Newly submitted systems contributed to the strongest per-accelerator scores of the round.
4. Hybrid and scalable deployments increased. The round included the first heterogeneous submission combining accelerators from different vendors and geographically distributed submissions spanning continents.
5. Inference continued to scale out. A 512-GPU submission set a new record for the largest-scale system in the benchmark.

The Closed division remains the basis for direct comparisons, while the Open division exposes optimization approaches that can indicate future mainstream techniques. Together, the results show how inference systems are evolving across models, software, accelerators, and deployment scales.