MLCommons Publishes MLPerf Storage v3.0 Benchmark Results

MLPerf Storage v3.0 expands coverage of AI storage workloads

MLCommons has released results from the MLPerf Storage v3.0 benchmark suite, which measures storage-system performance for machine learning workloads in an architecture-neutral, representative, and reproducible way.

Version 3.0 broadens the benchmark coverage to reflect the range of storage workloads generated by AI systems. It also introduces an S3 object storage access layer alongside the existing POSIX layer.

Two new benchmark tests

Version 3.0 adds a KV Cache test that measures storage performance for read and write operations involving large language model inference caches. KV caching is widely used to improve transformer-based inference performance, especially in autoregressive applications such as LLMs, which repeatedly access the same key-value vectors.

The release also introduces a Vector Database (VDB) test for storage performance in vector indexing and query workloads. Vector databases store high-dimensional data that can exceed the capabilities of traditional database architectures. AI applications commonly use them to store unstructured information, including text and media.

According to Brian Belgodere, co-chair of the MLPerf Storage working group, the new tests expand the suite's coverage of AI inference workloads that create storage requirements. The training tests cover training and checkpointing, while the inference tests now include KV caching and vector databases. By examining specific storage patterns instead of only complete AI systems, the benchmark provides more detailed information for engineering and provisioning systems while reducing storage bottlenecks.

S3 object storage access layer

Version 3.0 adds support for an S3 object storage access layer as an alternative to the existing POSIX-compliant layer. This allows performance comparisons across a broader range of hosted storage options using the same workload.

The S3 layer supports training, checkpointing, and some VDB tests in the v3.0 suite. About one-sixth of the submissions in this round used the S3 storage access layer.

Curtis Anderson, co-chair of the MLPerf Storage working group, said that support for S3 expands the range of storage systems represented in the benchmark. As AI contexts scale into the trillions, object-based storage could become a viable, and potentially preferred, alternative to filesystem-based storage. Including S3 support gives organizations additional performance information when selecting and combining storage technologies for specific applications.

Eleven organizations submit for the first time

Nineteen organizations submitted results in this benchmark round. The eleven first-time submitters were Azure, Everpure, HolmesAI, Nebius, NewFW, NVIDIA, OpenLake, Suzhou Zishan Longlin, TuringData, XSKY, and ZettaLane.

David Kanter, Head of MLPerf, said the submissions represent a broad cross-section of the storage community, including cloud and on-premises solution providers, as well as organizations developing storage systems and devices.

The results also include measurements of power efficiency. On-premises submissions for the checkpointing write test reached a median rate of 14 GB/second per watt, with a maximum of 201 GB/second per watt. For the UNet3D read test, submissions achieved a median of 34 GB/second per watt, with a maximum of 277 GB/second per watt.

Anderson noted that the results show a broad range of power efficiencies and indicate further potential for improvement in storage systems used for AI applications.

Five years of collaborative development

The MLPerf Storage benchmark was developed over five years through a collaborative engineering process involving 35 storage solution providers and academic research groups.

The open-source, peer-reviewed suite provides a common basis for competition and evaluation across the industry. It measures performance and energy efficiency while supplying technical information for organizations procuring and tuning AI training and inference systems.