First AA-AgentPerf Results for NVIDIA Vera Rubin NVL72
First AA AgentPerf Results for NVIDIA Vera Rubin NVL72 October 15, 2026 Initial AA AgentPerf results for NVIDIA’s Vera Rubin NVL72 show up to approximately 5.7 times the total t...
By Hardware Team
First AA-AgentPerf Results for NVIDIA Vera Rubin NVL72
October 15, 2026
Initial AA-AgentPerf results for NVIDIA’s Vera Rubin NVL72 show up to approximately 5.7 times the total throughput per megawatt of its predecessor, GB300 NVL72, on DeepSeek V4 Pro. The gains are approximately 5.4 times on DeepSeek V4.1 Flash, 3.0 times on GLM-5.2, and 2.8 times on Kimi K3.
Vera Rubin is NVIDIA’s successor to Blackwell. Vera Rubin NVL72 is a rack-scale system containing 72 Rubin GPUs and 36 Vera CPUs.
Largest throughput-per-megawatt leads over GB300 NVL72
Results as of October 15, 2026:
- Approximately 5.7 times: DeepSeek V4 Pro at approximately 190 tokens per second per user
- Approximately 5.4 times: DeepSeek V4.1 Flash at approximately 580 tokens per second per user
- Approximately 3.0 times: GLM-5.2 at approximately 165 tokens per second per user
- Approximately 2.8 times: Kimi K3 at approximately 67 tokens per second per user
These results and charts reflect AA-AgentPerf measurements available on October 15, 2026.
Agentic coding workloads are difficult to serve efficiently
Coding agents operate in a loop. The model reasons and calls a tool, the harness runs that tool and adds its output to the conversation, and the complete conversation is sent back to the model for the next step.
Because every request includes the preceding context, the input grows as the agent reads files, runs tests, and makes edits. The model’s replies generally remain short. This makes agentic traffic different from chat workloads and fixed-length prompts used in many inference benchmarks.
AA-AgentPerf replays coding-agent trajectories generated by agents resolving issues in public code repositories. It tests a range of concurrency levels, with each simulated agent processing three trajectories in sequence and pausing after each reply for a tool-call interval based on real tool-call durations.
Requests can contain up to 132K input tokens, while replies are often short tool calls. On average, 97% of a request’s input tokens repeat the previous request.
When a serving stack retains this repeated prefix in the KV cache, it only needs to prefill the tokens added during the current turn instead of reprocessing the entire context. In one longer trajectory, the context grows from 10K to 116K tokens over 118 requests. The agent sends 9.7M input tokens to the model in total, but only 116K are new. Across all replies, the model generates 20K output tokens.
Input tokens per request in one agent session
In this example, each request is divided into the prefix repeated from the previous request and the tokens added since then. The tokens are processed with o200k_base.
Agentic inference requires specific treatment during benchmarking because of several characteristics:
- KV-cache capacity and reuse: Thousands of concurrent agents can each maintain tens of thousands of context tokens. Those tokens must remain cached during tool-call pauses and be located again for the next request.
- Uneven prefill: Each turn can add a few tokens or tens of thousands of tokens, such as when a tool returns a large file, on top of a long cached prefix.
- Short outputs: Many replies are brief tool calls, making time to first token and scheduling overhead more significant than in long-form generation.
AA-AgentPerf’s workload includes the following ranges:
- Up to 132K input tokens per request, with 1M-token contexts planned for a future version
- Up to 182 requests in a single trajectory
- Up to 11M input tokens sent to the model across one trajectory
- Up to 69K new input tokens added in one turn when a tool returns a large output
- Tool-call pauses of up to 5 seconds between requests
- An average of 97% of input tokens repeated from the previous request
These workload statistics cover 500 trajectories and 18,997 requests, tokenized with o200k_base.
Vera Rubin NVL72 leads GB300 NVL72 by up to approximately 5.7 times per megawatt
Vera Rubin NVL72 retains the rack-scale design of GB300 NVL72, with 72 GPUs and 36 CPUs in a single NVLink domain.
Each Rubin GPU has the same 288 GB of memory as Blackwell Ultra, but offers higher bandwidth and compute in several areas. HBM4 provides 2.4 times the memory bandwidth, NVLink bandwidth is 1.7 times higher, and dense NVFP4 compute is 2.3 times higher. The Vera CPU also doubles NVLink-C2C bandwidth to its GPUs, which carries KV cache data offloaded to CPU memory.
| Specification per GPU | Rubin, Vera Rubin NVL72 | Blackwell Ultra, GB300 NVL72 |
|---|---|---|
| GPU memory | 288 GB HBM4 | 288 GB HBM3E |
| Memory bandwidth | 19.2 TB/s | 8 TB/s |
| NVLink bandwidth | 3.0 TB/s, NVLink 6 | 1.8 TB/s, NVLink 5 |
| Dense NVFP4 compute | 35 PFLOPS | 15 PFLOPS |
| Sparse NVFP4 compute | 50 PFLOPS | 20 PFLOPS |
| CPU | Vera, 88 cores | Grace, 72 cores |
| CPU-to-GPU link per superchip | 1.8 TB/s NVLink-C2C | 0.9 TB/s NVLink-C2C |
| Measured power in these results | 0.5 to 1.9 kW | 0.3 to 1.0 kW |
The specifications are from NVIDIA’s Vera Rubin NVL72, Blackwell Ultra, and GB300 NVL72 materials as of October 2026. Measured power refers to each GPU’s own power draw under load, including the GPU and its HBM. It excludes CPUs, networking, and cooling, and is based on NVIDIA’s DCGM/SMI telemetry across the configurations on each system’s frontier for the four models.
Results across the four models
The charts measure total throughput per megawatt of measured GPU power against per-user output speed. Total throughput includes input tokens, both cached and newly prefilled, and output tokens. Approximately 96% of the tokens are cached input.
Per-user speed is the 25th percentile, or p25, across requests and excludes time to first token. Each point represents a serving configuration at a particular number of agents.
At peak throughput, Vera Rubin NVL72 records the following results compared with GB300 NVL72:
- DeepSeek V4 Pro: 272M tokens/s per MW versus 239M, or 1.14 times as much
- DeepSeek V4.1 Flash: 571M versus 395M tokens/s per MW, or 1.45 times as much
- GLM-5.2: 272M versus 201M tokens/s per MW, or 1.35 times as much
- Kimi K3: 70M versus 56M tokens/s per MW, or 1.26 times as much
Vera Rubin’s lead is largest where GB300 throughput declines at higher per-user speeds. The differences reach approximately 5.7 times on DeepSeek V4 Pro at approximately 190 tokens/s per user, 5.4 times on DeepSeek V4.1 Flash at approximately 580 tokens/s, 3.0 times on GLM-5.2 at approximately 165 tokens/s, and 2.8 times on Kimi K3 at approximately 67 tokens/s.
On DeepSeek V4 Pro, increasing per-user speed from approximately 120 to 165 tokens/s reduces GB300’s throughput per MW by 81%, compared with a 16% reduction for Vera Rubin.
The difference is wider on a per-GPU basis. Vera Rubin provides a 2.0 to 2.5 times advantage at peak throughput, reaching approximately 12 times on DeepSeek V4 Pro at approximately 165 tokens/s per user, approximately 18 times on DeepSeek V4.1 Flash at approximately 580 tokens/s, approximately 7 times on GLM-5.2 between approximately 200 and 310 tokens/s, and approximately 7 times on Kimi K3 at approximately 176 tokens/s.
The fastest Vera Rubin configurations also record the highest per-user speeds on the benchmark for DeepSeek V4.1 Flash, at 906 tokens/s, GLM-5.2, at 408 tokens/s, and Kimi K3, at 255 tokens/s.
Higher power efficiency reduces the operating cost of serving each token. It also allows more agents to be served within the same datacenter power budget. At approximately 165 tokens/s per user, the same GPU power budget serves roughly five times as many DeepSeek V4 Pro tokens on Vera Rubin NVL72 as on GB300 NVL72.
DeepSeek V4 Pro shows the widest lead at high speeds
DeepSeek V4 Pro has 1.6T total parameters and 49B active parameters. It uses a combination of compressed sparse attention and heavily compressed attention to keep its KV cache small at long context lengths.
At peak throughput, Vera Rubin NVL72 serves 272M tokens/s per MW at 69 tokens/s per user. GB300 NVL72 serves 239M tokens/s per MW at 67 tokens/s per user, and Vera Rubin remains approximately 1.1 times ahead up to about 120 tokens/s per user.
The gap widens above approximately 130 tokens/s per user. At approximately 165 tokens/s per user, Vera Rubin serves 174M tokens/s per MW compared with 34M for GB300, a 5.1 times advantage. At approximately 190 tokens/s per user, the figures are approximately 132M and 23M, respectively, producing a lead of approximately 5.7 times.
NVIDIA’s GB300 rack configurations reach approximately 163 tokens/s per user. Above that speed, the GB300 line is based on smaller 4-GPU configurations. Vera Rubin’s fastest configuration reaches 298 tokens/s per user. Above approximately 255 tokens/s per user, those 4-GPU GB300 configurations provide higher throughput per MW than Vera Rubin and extend GB300’s results to 340 tokens/s.
Vera Rubin configurations have a longer time to first token. At approximately 165 tokens/s per user, the median is 1.65 seconds compared with 1.08 seconds for GB300. When measured using end-to-end speed, which includes the initial wait, GB300 matches or exceeds Vera Rubin’s throughput per MW at most speeds.
DeepSeek V4.1 Flash sets the highest throughput and speed results
DeepSeek V4.1 Flash has 552B backbone parameters. It activates 8B parameters per token during prefill and 16B during decode, and includes a 196B-parameter Engram memory accessed through token lookup.
Its sparse attention and FP4 KV cache reduce the KV-cache requirement to approximately 890 bytes per token, around one quarter of DeepSeek V4 Flash’s requirement.
Vera Rubin NVL72 reaches 571M tokens/s per MW at 68 tokens/s per user. This is the highest throughput per MW recorded across the four models. The result compares with 433M for B300 and 395M for GB300 NVL72, at 31 and 23 tokens/s per user, respectively.
Vera Rubin’s advantage over GB300 grows with per-user speed:
- Approximately 1.8 times at 68 tokens/s
- Approximately 2.9 times at approximately 165 tokens/s
- Approximately 3.4 to 3.6 times between approximately 230 and 350 tokens/s
- Approximately 5.4 times at approximately 580 tokens/s, using a 12-GPU configuration that spreads decode across five workers
The fastest Vera Rubin result is 906 tokens/s per user from a single 4-GPU server serving one agent. This is the fastest result for any model in the benchmark, ahead of 669 tokens/s on B200 and 596 tokens/s on GB300 for DeepSeek V4.1 Flash.
Per GPU, Vera Rubin reaches 651K tokens/s at peak, compared with 284K for GB300, and is approximately 6.5 times ahead of the GB300 line at approximately 300 tokens/s per user.
NVIDIA’s Vera Rubin runs capped DeepSeek V4.1 Flash replies at 2,048 output tokens, compared with 16,384 for the other DeepSeek V4.1 Flash results. Approximately 2% of replies reached the cap. The average reply was also about one-tenth shorter, at 275 to 301 tokens compared with 319 to 335 tokens, which modestly favors Vera Rubin’s throughput.
GLM-5.2 shows an approximately 3 times lead at higher speeds
GLM-5.2 has 753B total parameters and 40B active parameters, with DeepSeek-style sparse attention. GLM-5.2 and GLM-5.3 share an architecture and are shown on the same leaderboard chart.
Vera Rubin NVL72 reaches 272M tokens/s per MW at 84 tokens/s per user. GB300 NVL72 reaches 201M at 68 tokens/s per user, giving Vera Rubin a 1.35 times advantage at peak throughput.
At approximately 165 tokens/s per user, Vera Rubin serves 159M tokens/s per MW, approximately three times the GB300 result of about 53M. The advantage remains between approximately 2.7 and 3 times through approximately 310 tokens/s per user.
Vera Rubin’s fastest result is 408 tokens/s per user, the fastest GLM-5.2 result on the benchmark. It is ahead of 333 tokens/s on B300 and 330 tokens/s on GB300.
When measured using end-to-end speed, including time to first token, Vera Rubin still provides 1.1 to 1.8 times GB300’s throughput per MW across the measured range.
Kimi K3 shows an approximately 2.8 times lead at moderate speeds
Kimi K3 is the largest model in the results, with 2.8T total parameters and 104B active parameters. Its hybrid architecture uses linear attention in 69 of its 93 layers.
All systems serve fewer tokens per MW on Kimi K3 than on the other three models. Vera Rubin NVL72 peaks at 70M tokens/s per MW at 41 tokens/s per user, compared with GB300 NVL72’s peak of 56M at 40 tokens/s per user.
At approximately 67 tokens/s per user, Vera Rubin serves approximately 59M tokens/s per MW, about 2.8 times the GB300 result of approximately 21M. At 176 tokens/s per user, its 30-agent configuration serves 9.6M tokens/s per MW compared with approximately 4.0M for GB300, or approximately 2.4 times as much.
Current Vera Rubin configurations leave a gap between approximately 74 and 176 tokens/s per user.
Vera Rubin’s fastest result is 255 tokens/s per user from a single 8-GPU vLLM server. This is the fastest Kimi K3 result on the benchmark, ahead of B300 at 235 tokens/s and GB300 at 194 tokens/s.
Time to first token is long at peak throughput on both systems. The p95 is 9.3 seconds for Vera Rubin’s 2,048-agent configuration and 19 seconds for GB300’s 768-agent configuration.
NVIDIA tuning for agentic workloads
NVIDIA provided details about its Vera Rubin and GB300 configurations. The current configurations are similar between the two systems, so much of the performance difference comes from the newer rack generation’s hardware, along with early Rubin-specific kernels and tuning.
The techniques include the following.
KV caching across memory tiers
Because 97% of input tokens are repeated from the preceding request on average, prefix caching is essential. Deployments can use memory tiers from HBM to host memory, with disk potentially serving as a further tier.
Most Vera Rubin configurations offload KV cache to CPU memory over NVLink-C2C. Vera doubles this bandwidth to 1.8 TB/s per superchip, and transfers overlap other GPU work when agentic contexts expose GPU-to-CPU bandwidth in ways that short synthetic prompts do not.
DeepSeek V4 Pro prefill workers are configured with 120 to 200 GiB of host memory for KV cache. DeepSeek V4.1 Flash configurations instead keep the model’s 196B-parameter Engram memory in CPU memory rather than HBM.
Session-aware routing with NVIDIA Dynamo
Agent sessions return to the same context on every turn, so routing each request to the worker that already holds its cache can reduce repeated work.
Every Vera Rubin configuration runs behind NVIDIA Dynamo, an open-source serving framework. Dynamo keeps each agent’s turns on the same prefill worker. For DeepSeek V4 Pro, GLM-5.2, and Kimi K3, it also sends each request to the worker holding the largest portion of its prefix in cache.
Kimi K3’s linear-attention layers introduce an additional constraint. Their recurrent state cannot be rolled back to an earlier point like a standard KV cache, so reuse requires a snapshot saved at the appropriate location.
Disaggregated serving and wide expert parallelism
A turn can add a few tokens or tens of thousands of tokens. Except at the lowest concurrency levels, prefill therefore runs on separate workers so a large tool output does not delay decoding for other agents.
NVIDIA sizes the prefill and decode sides according to concurrency. For DeepSeek V4 Pro, this means one to six 4-GPU prefill workers as concurrency rises. The model’s experts are distributed across as many as 32 decode GPUs, and each request’s KV cache is transferred from prefill to decode over NVLink.
Kimi K3’s highest-throughput configurations also use DWDP, or distributed weight data parallelism, a TensorRT-LLM prefill technique described in a related paper.
Speculative decoding tuned to batch size
All four models use speculative decoding:
- DeepSeek V4 Pro and Kimi K3 use DSpark draft models
- DeepSeek V4.1 Flash uses built-in DSpark drafting
- GLM-5.2 uses its own multi-token prediction layers
Draft length is adjusted for each concurrency level. For example, DeepSeek V4 Pro uses between 7 draft tokens at lower concurrency and 3 at the highest concurrency. Speculation is generally most effective when batches are small and GPUs have spare compute capacity. The configuration behind the approximately 5.7 times result on DeepSeek V4 Pro uses 7 draft tokens.
Rubin-tuned kernels
NVIDIA’s NVFP4 checkpoints run mixture-of-experts layers in 4-bit precision using NVFP4 kernels tuned for Rubin. The configurations also use fused kernels for individual model architectures, including DeepSeek V4 Pro’s sparse attention and Hyper-Connections.
The same attention kernel must process inputs ranging from a few tokens to 1M tokens. Autotuning therefore needs to cover a range of shapes instead of specializing in fixed shapes, which is important for agentic workloads and is not exercised by many fixed-shape benchmarks.
Fast host-side request handling
Each request includes the entire conversation so far, up to 132K tokens in this workload. The serving frontend must tokenize and forward that history on every turn.
Dynamo’s router is written in Rust to reduce host-side latency. DeepSeek V4 Pro and GLM-5.2 configurations add a fast tokenizer with a tokenization cache and streamlined request transport. Configurations for the other models pin each worker to the CPU cores and memory nearest its GPUs.
These are the first Vera Rubin results. Further performance improvements may result from additional Rubin-specific optimizations and tuning.
Vera Rubin configuration overview
| Model | Serving stack | Weights | Prefill and decode | Speculative decoding | KV cache | Routing | GPUs per configuration |
|---|---|---|---|---|---|---|---|
| DeepSeek V4 Pro | TensorRT-LLM with NVIDIA Dynamo | NVIDIA NVFP4 checkpoint | Disaggregated. DEP4 prefill workers, wide expert-parallel decode using DEP16 or DEP32, and TP4/EP4 decode at 34 agents | DSpark, 3 to 7 draft tokens | FP8, offloaded to host memory on prefill workers | KV-aware, with session affinity | 32 to 56 |
| DeepSeek V4.1 Flash | vLLM with NVIDIA Dynamo | DeepSeek release checkpoint, MXFP4 | Aggregated TP4 or TP2 for 1 and 2 agents. Disaggregated from 64 agents, with one or two TP2 prefill workers and DEP4 decode, or five TP2 decode workers | Built-in DSpark drafting, 6 draft tokens | Kept in GPU memory, with the model’s Engram memory offloaded to host memory | KV-aware across two prefill workers at 1,024 agents, and a single prefill worker otherwise | 2 to 12 |
| GLM-5.2 | TensorRT-LLM with NVIDIA Dynamo | NVIDIA NVFP4 checkpoint | Disaggregated. One to three prefill workers using TP2/EP2 or DEP4, with decode on TP8, TP4/EP4, or DEP8 | MTP, 4 to 8 draft tokens | FP8, offloaded to host memory on prefill workers in all but one configuration | KV-aware, with session affinity | 10 to 20 |
| Kimi K3 | TensorRT-LLM and vLLM for 1 agent, both with NVIDIA Dynamo | NVIDIA NVFP4 checkpoint | Aggregated TP8 on vLLM for 1 agent. Disaggregated from 16 agents: a TP8 prefill worker with three TP8 decode workers at 16 and 30 agents, and DWDP16 prefill with one or two groups and DEP16 decode from 512 agents | DSpark, 3 to 7 draft tokens | FP8, offloaded to host memory in TensorRT-LLM configurations | KV-aware, with session affinity | 8 to 48 |
Further details on the benchmark
AA-AgentPerf measures total server throughput, including input and output tokens, per megawatt of accelerator power. Input tokens include KV-cache hits. Output speed is the p25 per-request decode rate after the first token arrives, excluding the wait for the first token.
The workload uses real coding-agent trajectories, includes tool-call pauses, and retains long repeated prefixes. This makes KV-cache reuse, routing, prefill scheduling, short-output performance, and time to first token central parts of the measurement. dove