MLCommons to Develop MLPerf Agents for NSF-Supported AI Efficiency Challenge

MLCommons has been selected as the benchmarking team for the STRIDE Ventures AI Efficiency Challenge, a program supported by the U.S. National Science Foundation Directorate for Technology, Innovation and Partnerships (NSF TIP). Through a $3.5 million award over two years, MLCommons will develop MLPerf Agents, a benchmark designed to measure how efficiently agentic AI systems complete real-world tasks, from receiving a delegated goal to producing a verified result.

Why agentic AI needs a new benchmark

Benchmarks that measure inference time, cost, and energy remain important, but agentic AI introduces additional factors that must be evaluated.

Unlike a single model responding to a single prompt, an agent can operate across multiple turns, call tools, retrieve information, manage expanding context, recover from failures, and verify its own work.

Existing MLPerf benchmarks already address parts of this complexity. The MLPerf Inference working group is developing agentic inference benchmarks that represent the demands agents place on model-serving systems, including multiple turns, tool calls, growing context, and cache reuse.

MLPerf Agents will assess the complete workflow. Its scope will include the agent harness, model selection, agentic inference, retrieval, tool execution, orchestration, state management, failure recovery, and validation. This broader approach will allow researchers and users to examine how both agent harnesses and models affect end-to-end efficiency, rather than measuring only the speed at which a model generates tokens.

What MLPerf Agents will measure

MLPerf Agents is planned to provide four main capabilities:

• Efficiency linked to successful outcomes: Cost, energy, latency, and throughput will be measured for tasks that are completed successfully. Success will be verified using techniques influenced by MLCommons AI Risk and Responsibility working groups.
• Whole-stack measurement: Resource use will be tracked across the complete agent stack to identify where inefficiencies occur.
• Comparable results across architectures: Shared workloads, tracing methods, measurement rules, and validation procedures will allow buyers to compare complete systems.
• Verification against benchmark gaming: Hidden validation datasets, together with MLPerf’s review and audit process, will help support reliable results.

Initial workloads are planned for software engineering, research and data analysis, and customer service. The benchmark is intended to provide buyers with evidence for selecting efficient agentic systems and help developers identify improvements that reduce cost and energy use as agentic AI expands.

Building on MLPerf

MLPerf Agents will build on eight years of MLPerf benchmarks used to compare AI systems, including MLPerf Inference and MLPerf Power. It will use the same open and collaborative process, with workloads, rules, and reference implementations developed by the MLCommons community and submitted results subject to peer review.

David Kanter, MLCommons founder and head of MLPerf, and Professor Vijay Janapa Reddi, co-founder and vice president of MLCommons, will lead the work with the MLCommons community. Qualcomm, NVIDIA, Intel, AT&T, Workato, Gimlet Labs, and other organizations will provide support.

As agentic AI scales, efficiency depends on more than model inference. The agent harness, model calls, tool invocation, retries, and related operations all contribute to cost and energy use. MLPerf Agents is intended to provide evidence that developers can use to identify these inefficiencies and that buyers can use when evaluating agentic systems.

About the AI Efficiency Challenge

The AI Efficiency Challenge is operated by STRIDE Ventures, a national platform for breakthrough innovation launched by the U.S. National Science Foundation and Start2 Group. Start2 Group administers the challenge under an Other Transaction Agreement with NSF.

This work is supported by the U.S. National Science Foundation. The opinions, findings, conclusions, and recommendations expressed in the related material belong to its authors and do not necessarily reflect the views of the U.S. National Science Foundation.