MLCommons Updates Its Mission to Standardize AI Measurement

MLCommons Updates Its Mission to Standardize AI Measurement

After eight years of significant growth and change in the AI industry, MLCommons is updating and clarifying its mission:

Standardize AI measurement and enable informed decisions.

Measurement is central to responsibly realizing AI's potential. AI systems remain complex black boxes that use patterns across billions of numbers to produce sophisticated behavior. Explanations of how these systems work are still limited, while their complexity continues to increase. Training and operating AI systems can also require substantial computing resources and energy. Better measurements help characterize both system behavior and operational costs empirically.

MLCommons aims to measure any characteristic of AI hardware, software, or data that affects delivered value, including speed, power consumption, reliability, and quality. These measurements support decisions about available AI systems, enable cost-benefit analysis, guide innovation, and contribute to risk management activities such as assurance and insurance. The work is intended to benefit academic researchers, hardware and software vendors, deployers, end users, and policymakers.

MLCommons describes this activity as a way to reduce risk and support growth in the AI market.

Standardized measurements

The organization focuses on developing and operating standardized measurements through industrial-quality benchmarks. These benchmarks combine detailed protocols with supporting artifacts, including test harnesses, held-out test datasets, and evaluation models.

Because AI testing can be complex and, in some cases, subjective, comparable results cannot necessarily be achieved through multiple implementations based only on a written standard. Some benchmark artifacts also require technical and legal protection to limit overfitting. In addition, measurements must evolve as the technology changes. These requirements call for a trusted organization that can operate sustainably.

MLCommons addresses these independent measurement needs while supporting the broader standards work of organizations such as ISO, Appia, CoSAI, and ADA.

The benchmark lifecycle

MLCommons's work includes defining, building, operating, validating, explaining, evolving, and connecting its benchmarks. Each activity contributes to the overall mission:

• Defining: Establishing what is being measured, whether an AI system under test or a dataset, helps make measurements standardized, understandable, and repeatable. Defining metrics involves translating goals into rules for protocols, harnesses, expected behaviors, measurements, and result presentation. MLCommons states that it emphasizes technically rigorous definitions developed through a transparent community process and does not engage in pay-for-play "benchmarketing."
• Building: Creating a benchmark requires careful engineering and, often, new scientific work. Real-world use cases must be translated into rigorous test scenarios while preserving their essential characteristics. Subjective benchmarks also require maintainable AI-based evaluations that keep pace with progress and continue to measure the intended task.
• Operating: Running a benchmark turns a code repository and dataset into an active measurement instrument. Infrastructure such as Candela and federated evaluation is intended to make benchmarks easier to use while maintaining measurement integrity.
• Validating: Third-party validation of benchmark results provides confirmation for performance claims. This can support purchasing decisions, assurance, insurance, and regulatory compliance. MLCommons aims to provide highly accurate measurements and support organizations that use them to define requirements.
• Explaining: Technical explanations help users understand and apply benchmark results. MLCommons provides commentary on the issues underlying its benchmarks, briefs experts who use the results, and works with standards and assurance organizations to incorporate findings into protocols and policies.
• Evolving: Continuous improvement helps benchmarks remain useful as systems become more capable. MLCommons monitors how measurements are used, tracks technological progress, and watches for emerging threats. Benchmarks must continue to reveal meaningful differences between tested systems as those systems improve.
• Connecting: Linking benchmarks with standards organizations and emerging policy frameworks around the world helps create an industry ecosystem. This is intended to help members deliver greater value to customers with lower risk and improved efficiency.

Planned organizational changes

In the coming days and months, MLCommons plans to make organizational changes intended to support this mission. These include:

• Serving results consumers more effectively: Improvements to the website and additional supporting resources will make results easier to find and use, including work associated with MLPerf Endpoints.
• Promoting benchmarks in new ways: MLCommons plans to add more product and marketing expertise to its board and staff and expand the Candela service.
• Supporting efficiency tradeoffs: Performance and reliability work will be aligned at both the organizational and technical-infrastructure levels. The goal is to help AI vendors and deployers deliver higher quality at lower cost and power consumption.
• Broadening the community: AI deployers that are currently members' customers will be encouraged to join as members. MLCommons also plans to revise its fee structure so that costs scale more proportionately with benefits.
• Expanding its scope: The organization will explore benchmarking in critical markets such as efficiency, healthcare, and financial services.

MLCommons expects its work to continue evolving as AI advances, including in emerging areas such as AI data science. The organization describes member participation and support as essential to measuring AI and informing the industry's development and adoption by society.