MLPerf Introduces an End-to-End RAG Inference Benchmark

MLCommons' MLPerf Inference Working Group has introduced the first version of a new end-to-end Retrieval-Augmented Generation (RAG) benchmark.

RAG answers questions using documents retrieved at query time instead of relying only on information stored in model weights. This approach can reduce hallucinations and provide access to current, private knowledge, making RAG a common architecture for deploying language models. In production, a RAG system is typically a pipeline of several models that retrieve relevant documents and reason over them to produce a grounded answer.

A RAG pipeline ingests and chunks source data, converts it into semantic vector embeddings for storage and retrieval, and combines retrieved information with a user query to generate a response. The benchmark measures this complete pipeline through two workloads:

• Ingestion: Builds a vector database from a document corpus.
• Question answering (QnA): Answers queries over the vector database by repeating retrieval and reasoning steps across multiple hops until sufficient evidence is available.

Together, these workloads represent a real RAG deployment, which may serve multiple models with different roles and sizes through an iterative loop. A single-model benchmark cannot measure these interactions.

This is the first multi-component MLPerf Inference benchmark to score an entire RAG pipeline end to end. It covers the workload, dataset, models, metrics, and reference implementation.

Why End-to-End RAG?

RAG is increasingly used in deployments where answers must be grounded in documents retrieved at query time. It can help reduce hallucinations, keep responses current, and allow general-purpose models to use proprietary or private information that was not part of their training data.

RAG is inherently a multi-component system. Its behavior depends on how the components are composed and served together, rather than on any single model. Single-model LLM benchmarks generally score one model against one prompt, often without measuring the tokenizer and detokenizer. As a result, they do not capture the pipeline behavior or the optimization opportunities involved in serving several models together.

The benchmark also provides a foundation for future agentic AI benchmarks. RAG is one of the tools an agent may use, and its multi-component, multi-model serving requirements provide an initial example of the challenges a broader agentic benchmark would need to measure.

The pipeline exposes optimization areas that do not arise in single-model benchmarks, including:

• Model placement: Mapping components across the system, such as placing smaller and larger language models on separate accelerators or running lightweight components on CPUs at different precision levels while meeting end-to-end accuracy requirements.
• Co-residency: Sharing one device among multiple models through memory partitioning or hardware-level isolation.
• Multi-stage scheduling: Overlapping concurrent tasks across heterogeneous accelerators using macro-batching and micro-batching throughout the pipeline and within individual components.
• Prefix caching: Reusing the shared context that grows across hops, trading compute-bound prefilling for memory-bound operations.
• System-level optimization and KPIs: Coordinating CPU, GPU, NIC, and storage resources to represent system-level performance in real-world deployment scenarios.

Dataset and Task Selection

The benchmark evaluates multi-hop question answering over Wikipedia documents using the FRAMES benchmark. Its data includes:

• 824 queries, each with a ground-truth answer and the Wikipedia article URLs needed to answer it.
• 2,515 Wikipedia HTML articles in a frozen snapshot, ensuring that every submission indexes the same corpus.
• Approximately 107,000 passages, created by splitting the articles into 768-character passages with 32 characters of overlap.

FRAMES was selected because it is public and factual, has unambiguous ground truth, and includes several reasoning types: numerical, tabular, temporal, multiple-constraint, and post-processing questions.

Its multi-hop queries are especially demanding because they cannot be answered from a single passage. The system must connect facts that do not appear together, requiring every stage of the pipeline to contribute.

For example:

“The person who posted a photo with Rahul Ligma and Daniel Johnson at the headquarters of a social media company claims to have a certain syndrome, despite never receiving a formal diagnosis. Who was this syndrome named after?”

Ground truth: Hans Asperger

The answer is not contained in one passage. The pipeline must connect three facts across separate articles: Elon Musk posted the photo, he said he had Asperger's syndrome, and the syndrome was named after Hans Asperger. The system must reformulate searches across multiple hops until the evidence connects.

The benchmark exercises this dataset through two independent pipelines, each treated as its own MLPerf workload:

• Ingestion (e2e-rag-db): Runs once to build the database.
• QnA (e2e-rag-qna): The scored end-to-end pipeline.

Ingestion Pipeline

The ingestion pipeline converts the corpus into a searchable vector database. This is a one-time operation.

Parsing extracts the article body from the 2,515 Wikipedia HTML files included with the benchmark and removes metadata such as references and navigation. Tables and lists are flattened into text row by row instead of being discarded.

Chunking divides the extracted text into 768-character passages. Each passage retains its original Wikipedia URL so it can be traced to its source page. The chunk size was selected to balance unrelated material in larger passages against the risk of separating related facts in smaller ones. A 32-character overlap helps preserve facts that fall near chunk boundaries.

Embedding converts each passage into a 768-dimensional vector. The vectors are then indexed in a FAISS HNSW graph for fast approximate similarity searches. The result is the vector database queried by every QnA run.

QnA Pipeline

The QnA pipeline answers queries against the database created during ingestion. Answering a query requires a task involving several components in an iterative loop, rather than a single model call.

1. A query rewriter decomposes the original query into up to three focused sub-queries.
2. The same embedder used during ingestion converts each sub-query into a vector.
3. The system retrieves the most similar passages from the vector database.
4. A reranker removes duplicate candidates and orders the remaining passages by relevance.
5. A document grader evaluates each passage and keeps those that help answer the query.
6. A sufficiency checker determines whether the accumulated evidence is enough.
7. If the evidence is insufficient, the query rewriter creates new sub-queries, and the process repeats for up to five hops.
8. When sufficient evidence is found, or the hop limit is reached, answer generation produces a response grounded in the retained passages. If the evidence is inadequate, the system returns “Unknown.”

Model Selection

The models are available through MLCommons-Storage.

• intfloat/e5-base-v2 serves as the embedder. It was trained for retrieval using separate query and passage encodings and has 110 million parameters.
• ColBERTv2.0 is the reranker. Its late-interaction approach performs token-level matching rather than compressing a passage into a single vector.
• GPT-OSS-120B handles query rewriting, evidence sufficiency checks, and final answer generation.
• GPT-OSS-20B grades retrieved documents. Its smaller size is intended for the high-volume relevance-classification stage.
• Llama-3.1-8B-Instruct serves as the judge. It comes from a different model family than the GPT-OSS models being evaluated, reducing the risk of self-preference bias.

Performance Metrics

MLPerf Inference traditionally defines Offline and Server serving scenarios. This first version focuses on Offline, in which all requests are available at once and can be scheduled for maximum throughput. Server support is planned for future rounds.

Each pipeline reports its own throughput metric:

• Ingestion: Documents per second.
• QnA: Tasks per second.

Documents per second is used for ingestion because each document passes through parsing, chunking, embedding, and indexing. It is the fixed unit of work for every submission, making results comparable across systems. Additional details are provided in the inference rules.

Tokens per second is not suitable for QnA because the pipeline combines two language models of different sizes with non-language-model components. A per-hop metric is also difficult to interpret because each task may require a different number of hops. Tasks per second instead measures the complete unit of work: one query answered from beginning to end.

The QnA workload can involve up to five hops and a dozen or more language-model calls for one query. It is therefore a complex and dynamic workload, made harder to reproduce by the nondeterministic nature of language-model generation.

For performance runs, the benchmark provides recorded reference inputs for every stage of every hop. These inputs fix the number of hops and the retrieved documents. Outputs are still generated normally and then discarded, allowing the pipeline to perform its work while minimizing run-to-run variation.

Accuracy Metrics

Retrieval quality against the required Wikipedia URLs is not an official metric. Instead, it serves as a database-integrity check to confirm that an independently built vector database behaves like the reference database and that submissions answer from the same corpus.

Reference values across all 824 queries are:

A submission is valid when its answer accuracy reaches at least 97% of the reference accuracy.

Accuracy varies according to the reasoning required. The following breakdown is provided for insight, and a query may belong to more than one reasoning category:

Multiple-constraint queries have the highest accuracy because they primarily require collecting discrete facts, a task suited to dense retrieval and language models. Numerical and tabular queries are more difficult because they require extracting exact figures from prose or tables and then performing calculations. A misread number or a chunk boundary that divides a table can cause the entire query to fail.

Temporal and post-processing queries fall between these groups. They require date calculations or a final transformation after correct retrieval.

Potential ways to address these weaknesses include knowledge graphs or structured indexes for querying entities and relationships, table-aware parsing and chunking, and tools such as code interpreters for arithmetic and unit conversion. These capabilities would move the system beyond a fixed retrieve-and-read loop toward one that chooses how to find and process information.

Compliance

E2E-RAG-QnA requires one compliance test, TEST09, which verifies output-token length. The test reruns the workload in LoadGen performance mode using audit.config and compares the mean output-token length from the answer generator with the reference value.

The reference implementation has a mean output sequence length of 273.81 tokens, with an allowed range of 246.43 to 301.19 tokens, representing a ±10% band. This check helps prevent a submission from reducing work in performance mode by systematically generating shorter answers than the reference.

The test is a single aggregate check on one pipeline stage. It does not cover retrieval, reranking, or intermediate query-rewriting and sufficiency-checking calls. Future revisions could add distribution-level or per-component compliance coverage.

Optimization Opportunities

The E2E RAG pipeline is designed to expose optimization opportunities across the inference stack, from vector database generation and algorithmic criteria to serving frameworks, dense and sparse kernels, and KV-cache-aware scheduling. The reference implementation provides a clear but unoptimized baseline in several areas.

• CPU pipeline: In the Offline scenario, MLPerf LoadGen provides all 824 queries or tasks at once to the serving framework. Submitters can explore task distribution across CPU cores, affinity to accelerators, memory placement, and coordination between the overall pipeline and its individual components.
• GPU partitioning and KV-cache balancing: Multi-model placement can be optimized to reduce latency and keep participating accelerators saturated across compute, communication, and storage. This includes CPU-GPU interaction, GPU-GPU or accelerator interaction, and balancing compute against communication across macro-batches and micro-batches.
• Multiple models with different precisions: Selected models can run at different precision levels as long as the end-to-end accuracy requirement is met. This allows submitters to choose precision selectively for system-level performance.

Conclusion

MLPerf's first end-to-end RAG benchmark measures a complete RAG pipeline rather than evaluating a single model in isolation. It captures the features that govern deployed RAG systems: multiple models with different roles and sizes, served together through an iterative, multi-hop process.

The benchmark also establishes groundwork for future agentic RAG and agentic AI evaluations. Its retrieve, reason, and decide loop resembles the loop used by agents, while capabilities such as tool use, structured retrieval, and greater autonomy in information processing could extend the benchmark toward a full agentic system.

References

• E2E-RAG reference implementation and getting started guide
• MLPerf Inference benchmark data
• A new GPT-OSS benchmark and DeepSeek R1 updates for latency-optimized reasoning
• MLPerf Inference rules for E2E-RAG
• FRAMES paper