MLPerf Training Adds Its First LLM Post-Training Benchmark

MLPerf Training will introduce an LLM post-training benchmark in the v6.1 submission round in October 2026. The benchmark complements the existing suite of pre-training tests.

Pre-training builds a model’s foundational capabilities through large-scale data processing. Post-training refines those capabilities for specific tasks. Since the second half of 2025, advances in post-training have contributed to smaller models approaching frontier-level performance, including Qwen 3.8 27B, as well as generational improvements to existing base models, such as those described for GLM-5.3.

The MLPerf Reasoning Task Force developed a workload that completes a real-world LLM post-training task in fewer than 1024 GB300 hours. Real-world post-training can combine distillation, reinforcement learning, and supervised fine-tuning across areas such as mathematics, reasoning, and agentic tasks.

After reviewing available methods, the task force selected Reinforcement Learning with Verifiable Rewards (RLVR), which is well suited to software tasks. An LLM-powered coding agent works on software-engineering problems in a sandbox environment. Each attempt is called a rollout. For every problem, the system samples multiple rollouts and assigns each a binary pass or fail reward. These traces are then used in Group Relative Policy Optimization (GRPO), which compares rewards within each group and updates the model so that higher-reward solutions become more likely. The updated weights are used for the next rollout iteration.

Developing the benchmark required the most complex system-level integration in MLPerf Training to date. It deploys separate training, inference, and agent environments. During execution, hundreds of concurrent rollouts compete for CPU time, GPU time, memory, and storage. These resources must be balanced, while the numerical behavior of modern training techniques must remain stable.

Model Selection

The benchmark uses Qwen 3.5 397B, the largest model in the Qwen 3.5 family, released in February 2026. It is an open-weight model distributed under the Apache 2.0 license.

Qwen 3.5 is a mixture-of-experts (MoE) model with 397 billion total parameters and 17 billion active parameters per token. It was the first model family to productize a hybrid architecture combining Gated DeltaNet (GDN) with sparse MoE layers. Each MoE layer activates one shared expert and 10 of 512 routed experts, or 2% of the routed experts.

GDN is a time- and space-efficient alternative to standard attention. It uses a fixed-size, compressed state instead of a linearly growing key-value cache. This approach conceptually learns past associations rather than storing them explicitly, and the fixed state is particularly useful for long contexts. Both MoE and GDN represent current architectural approaches for large language models.

MLPerf Training measures the time required to reach a predetermined accuracy target. This time-to-train metric prevents the benchmark from becoming a raw throughput test. A successful system must be both correct and fast.

Throughput-only benchmarks can encourage optimizations that improve speed while reducing final model quality. Reduced-precision numerical formats, for example, may increase throughput without providing enough numerical stability to reach the target accuracy. In asynchronous training, increasing policy lag can also improve throughput for one system while degrading model quality. AI benchmarks therefore need to measure trained model quality as well as throughput.

Applying this approach to post-training is difficult because large open-weight models have often already been post-trained on most publicly available datasets. The published Qwen3.5-397B-A17B weights already provide capability on many agentic tasks. This benchmark therefore targets a narrower objective: improving performance specifically on software-engineering tasks.

Dataset Selection

The benchmark uses the Apache 2.0-licensed R2E-Gym dataset of software-engineering problems. R2E-Gym contains procedurally generated executable environments based on GitHub commits from Python projects including Aiohttp, Datalad, NumPy, Pandas, Pyramid, Pillow, Sympy, and Tornado.

Each task provides the agent with a Linux container, a Git repository, pre-installed dependencies, and an issue description. The agent must produce a patch that resolves the issue.

To establish a stable operating range, the task force limited the maximum number of agent turns and the available context length. It then selected a subset of R2E-Gym using difficulty indicators such as the number of modified lines and files in the golden solution, along with the ability of the post-trained model to solve each task. The resulting dataset contains 700 training problems and a separate validation set of 251 problems.

The post-training package includes prompts and container images stored in Apptainer SIF format on MLCommons storage. Preparing the environments introduced architecture-related challenges. The original R2E-Gym images were built for x86_64 in early 2025. Rebuilding them for Aarch64 in 2026 produced small behavioral differences caused mainly by dependency drift, updated packages, and architecture-dependent packages.

Some problems consequently became unsolvable or behaved differently across architectures. The dataset was audited, and issues were fixed where possible. Problems that could not be made consistent were removed so that the remaining environments behaved the same way across architectures and remained solvable.

Environment

The model interacts with each sandbox through a harness. Given a context limit and turn limit, the harness influences how efficiently the model uses those budgets. The task force found that harness selection had a strong effect on solve rate, which directly affects training performance because RLVR uses sparse binary rewards.

The benchmark uses the OpenHands harness for both training and validation. During reinforcement learning, agents can sometimes obtain a reward without solving the underlying task. For example, an agent might remove failing tests instead of fixing the defect that caused them to fail. This behavior, known as reward hacking, can teach the model solutions that score well during training but fail to generalize. To reduce this risk, evaluation uses test files that the agent cannot see or modify during execution.

The benchmark sets a maximum context length of 65,536 tokens, a maximum of 30 agent turns, and 16 generations per prompt. These limits balance training performance and runtime. A larger context consumes more resources and can improve model performance, making quality improvements harder to achieve. A smaller context can prevent the agent from solving tasks, reducing rewards and limiting the training signal available to GRPO.

The objective is therefore to optimize the model’s ability to work effectively within a limited context and turn budget.

The model runs in instruction-following mode with reasoning mode disabled. Through the OpenHands harness, it can access a think tool. Long, multi-turn rollouts place substantial pressure on the key-value cache management systems used by generation endpoints. Implementations must balance retaining cached context in GPU memory against rerunning prefill operations for previously processed context.

Implementation Details

The reference implementation is built on NVIDIA NeMo-RL. It separates worker nodes into policy-training workers and rollout-generation workers, while agent sandboxes run on the same GPU nodes. Ray coordinates the distributed components:

• Megatron Bridge and Megatron Core train the policy.
• vLLM generates trajectories and receives refreshed policy weights after each training step.
• NeMo Gym runs the multi-turn OpenHands software-engineering agent in per-task Apptainer containers.

The reference implementation uses asynchronous training, allowing training and generation to overlap. This can result in generation using stale, or off-policy, weights. In the reference configuration, the maximum age of trajectories used for training is one weight version, although submissions may use different values.

The disaggregated design illustrates a central challenge in asynchronous reinforcement learning: training and inference must be rate-matched and optimized together. Improving only one part of the pipeline shifts the bottleneck to another part.

Evaluation Approach

Model performance showed substantial variability even with deterministic sampling at a temperature of 0.0. A model could solve a task on one attempt and fail on another. Multi-turn agentic trajectories are not fully reproducible because of tool calls, timing, and batching effects.

The benchmark uses pass@4 to make evaluation more stable. Each problem is evaluated four times, and the metric records the percentage of problems solved at least once. The validation set therefore produces 1,004 rollouts.

The target is a pass@4 score of 0.69, meaning that the model must solve at least once on 69% of the validation problems across four attempts. Validation sampling uses a temperature of 0.1 and top_p=0.95.

The task force compared pass@4 with pass@1 averaged across four attempts and pass^4, where all four attempts must succeed. Pass@4 had the lowest coefficient of variance.

Managing Variance and Evaluation Cost

Agentic reinforcement learning is less deterministic than pre-training. The training data is dynamic, and a feedback loop continuously updates the weights used for future rollouts. At the trajectory level, small numerical differences can change sampled tokens, which can alter tool calls and intermediate results. These differences compound across many turns.

Other sources of variability include request routing, differences between training and generation backends, numerical sensitivity in MoE routing, nondeterministic kernels, and off-policy training. Such variability is expected in an exploration-based learning process, but it complicates the use of repeatable quality targets.

Evaluation is also expensive. Unlike a forward-only validation pass, each evaluation launches complete software-engineering agents that generate tokens, use tools, and execute tests. In the reference configuration, one validation requires 1,004 trajectories and can consume several training steps.

The benchmark addresses these issues in three ways:

1. It uses grouped pass@4 to make the quality metric more stable.
2. It fixes the traversal order of the training dataset using a difficulty-aware ordering.
3. It delays the first full evaluation until the model is expected to be close to the target.

For qualified global batch sizes of 256, 512, and 1024, the reference configuration begins validation after 4,608, 5,120, and 7,168 training samples, respectively. Subsequent steps are evaluated as well.

These starting points follow an empirical scaling law derived from the reference convergence study:

GBS * ceil(2.5 + 3840 / GBS)

The schedule is designed so that runs reach the target at the first evaluation.

Benchmark Parameters

The benchmark reports the wall-clock time required to reach the fixed pass@4 target. The reference convergence study includes global batch sizes of 256, 512, and 1024, with batch-dependent learning rates, gradient-norm clipping, and first-validation steps.

The study ran on GB300 NVL72 racks using 64 compute nodes, each equipped with four Blackwell Ultra GPUs, for a total of 256 GPUs. Both training and generation used BF16. Each run required between 140 and 160 minutes, providing a baseline that submissions are expected to improve.

These settings define the qualified reference points. Submitters may choose other global batch sizes, provided they follow the MLPerf Training rules covering legal configurations, optimizations, and alternative implementations.

What the Benchmark Adds

The LLM post-training benchmark combines several systems concerns in one measured workload:

• Large-scale mixture-of-experts policy training
• High-throughput inference
• Frequent policy-weight transfers
• Asynchronous trajectory collection
• Long, multi-turn trajectories with intensive key-value cache use

This combination creates optimization challenges across computation, communication, scheduling, rollout throughput, environment execution, and the allocation of resources between training and generation. Implementations must also maintain numerical stability and avoid training collapse.

Because the benchmark measures the complete time-to-quality path, a successful optimization must accelerate the system without reducing the agent’s ability to solve unseen tasks.

Conclusion

MLPerf Training’s LLM post-training benchmark measures a form of model development that extends beyond next-token prediction or adaptation on a static dataset. The workload requires a model to learn how to take effective actions on real software-engineering problems.

By defining a common model, task, dataset, quality target, and evaluation process, the benchmark provides a standardized way to measure agentic reinforcement-learning systems. It covers the full post-training pipeline and identifies where time and resources are used.

The benchmark specification and reference implementation are available in the MLCommons Training repository on GitHub.