Nvidia Rubin CPX: A Prefill-Optimized Accelerator for Disaggregated Inference

Nvidia has announced the Rubin CPX, an accelerator designed specifically for the prefill phase of large-language-model inference. Its single-die design emphasizes compute throughput over memory bandwidth. The announcement follows Nvidia’s March 2024 introduction of the GB200 NVL72 Oberon rack-scale system and advances the idea that prefill and decode should run on hardware optimized for their different workloads.

The Rubin CPX

LLM inference consists of two main stages:

• Prefill, which processes the user’s prompt and generates the first token. It is generally compute-bound and determines time to first token (TTFT).
• Decode, which generates subsequent tokens while repeatedly loading data from the KV cache. It is generally memory-bandwidth-bound and affects time per output token (TPOT).

Prefill processes many tokens in parallel, so it can use substantial compute while leaving much of an accelerator’s memory bandwidth idle. Using an expensive GPU equipped with large quantities of high-bandwidth memory (HBM) for this workload can therefore leave part of the system’s cost underutilized.

The Rubin CPX addresses this imbalance with a compute-focused design and less expensive GDDR7 memory. It provides 20 PFLOPS of dense FP4 compute, or 30 PFLOPS using Nvidia’s sparse FP4 measurement, together with 128GB of GDDR7 memory and 2TB/s of memory bandwidth.

By comparison, the dual-die R200 provides 33.3 PFLOPS of dense FP4 compute, 50 PFLOPS sparse, 288GB of HBM, and 20.5TB/s of memory bandwidth. The CPX therefore offers less memory bandwidth but a relatively high level of compute for a single die.

Rubin CPX is a monolithic SoC packaged using conventional flip-chip BGA technology. Its 128GB of GDDR7 memory is expected to use a 512-bit bus operating at 32Gbps, producing the stated 2TB/s bandwidth. Moving from HBM to GDDR7 reduces memory cost per gigabyte by more than 50%.

The design also avoids HBM-related CoWoS packaging and does not require I/O chiplets. Its architecture is closer to a large monolithic graphics processor such as a next-generation RTX 5090 or RTX PRO 6000 Blackwell. The CPX is expected to deliver approximately 60% of the compute throughput of the R200, compared with roughly 20% for consumer-derived Blackwell GPUs relative to the B200.

Peak theoretical throughput will not necessarily be sustained. The estimated Rubin CPX power rating is approximately 800W, while a complete module containing GDDR7 memory may reach 880W. The design is also expected to face thermal constraints because the board uses a dense, sandwiched form factor.

Unlike the R200, Rubin CPX does not include NVLink SerDes for scale-up connectivity. It communicates with other GPUs through PCIe Gen 6 and CX-9 NICs on the scale-out network. Pipeline parallelism can use this lower-bandwidth connectivity because it primarily requires sequential send-and-receive operations rather than the all-to-all traffic associated with expert parallelism.

Memory as a System Constraint

Memory capacity has been important for loading larger models, while memory bandwidth has constrained training and inference token throughput. HBM capacity and bandwidth per GPU have increased rapidly, from 80GB and 3.4TB/s for the H100 to 288GB and 8.0TB/s for the GB300. This represents more than three times the capacity and approximately 2.5 times the bandwidth in less than three years.

HBM has consequently become a larger share of accelerator bill of materials (BOM). In the GB300, it is the largest single package component. HBM remains valuable for both training and inference, but its value differs between prefill and decode.

During decode, memory bandwidth is heavily used because each new token requires loading earlier tokens from the KV cache. During prefill, parallel processing reduces the relative demand for memory bandwidth, leaving some of the HBM capability unused. Because HBM carries a significant cost premium over other DRAM types, this underutilization increases the effective cost of prefill.

The Rubin CPX is intended to reduce that mismatch by combining high compute throughput with a smaller quantity of lower-cost memory.

Vera Rubin Rack Configurations

The Rubin CPX expands Nvidia’s VR200 rack-scale product family into three configurations:

1. VR200 NVL144: 72 GPU packages distributed across 18 compute trays, with four R200 GPU packages per tray.
2. VR200 NVL144 CPX: 72 logical GPU packages and 144 Rubin CPX GPUs across 18 compute trays, with four R200 packages and eight Rubin CPX packages per tray.
3. Vera Rubin CPX Dual Rack: One VR200 NVL144 rack paired with a separate VR CPX rack containing 144 Rubin CPX GPUs across 18 compute trays, with eight Rubin CPX GPUs per tray.

The corresponding compute-tray SKUs are:

• VR NVL144, containing Rubin GPUs
• VR CPX, containing Rubin CPX GPUs
• VR NVL144 CPX, combining Rubin and Rubin CPX GPUs

Vera Rubin is the third generation of Nvidia’s Oberon rack architecture, following the GB200 NVL72 and GB300 NVL72. The Vera Rubin systems are expected to become available in 2026 and introduce substantial changes to power delivery, cooling, cabling, and tray design.

The VR NVL144 CPX rack is liquid-cooled and has an estimated power budget of approximately 370kW, compared with approximately 190kW for the VR NVL144. The dual-rack configuration allows CPX capacity to be added separately to an existing VR NVL144 deployment. Because the VR CPX rack does not use NVLink or NVSwitch trays, it connects to the wider cluster through scale-out InfiniBand or Ethernet networking and does not need to be physically adjacent to the VR NVL144 rack.

The dual-rack design also allows operators to adjust the ratio of prefill to decode capacity independently. It can be deployed where infrastructure cannot support the approximately 370kW requirement of the combined rack, and separating the systems can reduce the impact of a rack-level failure.

Compute Tray Design

Each VR NVL144 CPX compute tray contains 22 Nvidia chips, including 14 accelerator or processor devices. Across an entire rack, this results in 396 Nvidia chips.

The design uses a cableless, modular layout to fit the required components into a 1U tray. The rear half retains a compute-board arrangement similar to that of the GB200 and GB300. One change is the use of socketable SOCAMM DRAM modules instead of soldered LPDDR5X for CPU memory.

The front half is organized around seven daughter-card modules:

• Four side-mounted daughter cards, two on each side, each containing two 800G CX-9 NICs, one 1.6T OSFP cage, one E1.S NVMe SSD module, and two Rubin CPX GPUs.
• A central daughter card containing the Bluefield-4 module, one Grace CPU, and one CX-9 NIC.
• A daughter card above the Bluefield-4 module containing the power delivery board (PDB). The PDB converts 48-54V from the rear busbar connector to 12-13.5V inside the chassis.
• A smaller utility-management module beside the Bluefield-4 module, containing components such as the BMC, HMC, DC-SCM, and management I/O.

The Rubin CPX modules at the front of the tray account for approximately 7,040W in total. This requires liquid cooling rather than the air cooling used in the corresponding portion of earlier designs.

Nvidia uses a sandwiched arrangement in which Rubin CPX and CX-9 daughter cards sit on opposite sides of a shared liquid-cooled cold plate. Heat pipes and spreaders transfer heat from the rear of the GDDR7 modules to the cold plate. Using both sides of the plate and the full 1U height increases component density.

The daughter cards are serviceable modules that slide into and out of internal rail-mounted bays. Signals travel from the HPM, also known as the Bianca board, through Amphenol Paladin board-to-board connectors to a PCB midplane. Daughter cards connect to the opposite side of that midplane through another set of Paladin connectors.

The CX-9 NICs are moved toward the front of the chassis, closer to the OSFP cages. This allows the longer PCIe Gen 6 connections to be routed over the PCB. PCIe Gen 6 signals operate at 64Gbit/s per lane in one direction, lower than the 200Gbit/s-per-lane Ethernet or InfiniBand signals used in earlier layouts. Higher-performance PCB materials are still needed to maintain signal integrity.

The CX-9 also acts as an integrated PCIe switch, making it important to the Rubin CPX topology and its scale-out connections.

Disaggregated Inference

Running prefill and decode on the same hardware creates competing resource demands. Prefill can occupy compute resources while leaving memory bandwidth unused, whereas decode can consume memory bandwidth while leaving compute capacity idle. Processing both phases concurrently can also make their performance interfere with each other.

Separating the phases onto different compute units, known as disaggregated serving, makes performance and service-level targets easier to manage. However, using identical hardware for both phases does not eliminate the underlying mismatch. A system running only prefill still contains more memory capacity and bandwidth than that workload requires.

The Rubin CPX applies a second level of specialization. It assigns prefill to an accelerator with high compute throughput and relatively modest memory bandwidth, while the R200 remains suited to decode workloads that benefit from HBM capacity and bandwidth.

The system includes 144 Rubin CPX chips, each providing 2.0TB/s of memory bandwidth, and 72 R200 chips, each providing 20.5TB/s. Together, they provide approximately 1.7PB/s of total system memory bandwidth.

In an illustrative total-cost-of-ownership analysis, running the same prefill workload on an R200 instead of a Rubin CPX produces approximately $0.90 per hour in wasted cost attributed to underused memory resources. The CPX does not eliminate bandwidth underutilization, but it places less expensive GDDR7 capacity behind the prefill workload.

The use of GDDR7 also reduces HBM’s share of total system content. For a fixed spending level on AI infrastructure, a greater use of CPX systems means that a smaller portion of that spending goes toward HBM. At the same time, lower token costs could increase demand for inference and therefore increase demand for decode capacity.

Pipeline Parallelism and Connectivity

Rubin CPX provides 16 lanes of PCIe Gen 6 for off-chip I/O, approximately 1Tbit/s of unidirectional bandwidth. The R200 provides 14.4Tbit/s through NVLink.

The lower-bandwidth interface is suitable for pipeline parallelism, in which different GPUs process successive layers of a model and pass activations from one stage to the next. For example, DeepSeek V3 in the NVFP4 format requires approximately 335GB of memory to hold all model weights, exceeding the 128GB capacity of a single CPX. Pipeline parallelism can distribute those layers across multiple GPUs.

Pipeline parallelism generally offers higher token throughput per GPU than expert parallelism because it uses simpler point-to-point communication. Its tradeoff is a higher time to first token, since activations move sequentially through the pipeline. Expert parallelism can provide lower latency in some cases but requires all-to-all collective communication.

For DeepSeek using PP8 or PP4, the estimated message size is 7kB per token. Fully saturating PCIe Gen 6 x16 would support a communications limit of approximately 18.3 million tokens per second. The estimated compute requirement is 0.074 TFLOP per token. Using the Rubin CPX’s stated dense FP4 throughput of 19,800 PFLOPS produces a compute-bound estimate of approximately 267.6 thousand tokens per second, well below the communications limit.

This means PCIe Gen 6 can provide sufficient connectivity for the described prefill workload, while NVLink would offer considerably more bandwidth than necessary. The estimated end-user cost of NVLink scale-up infrastructure, including NVSwitches and the backplane, is approximately $8,000 per GPU, slightly more than 10% of total cluster cost per GPU.

Expert parallelism would place much greater demands on the interconnect. DeepSeek V3 uses a top-k value of 8 and has 61 layers, so a simple estimate indicates that its communication requirements could be approximately 488 times higher than with pipeline parallelism.

Precision and Sparsity

The analysis of Rubin CPX focuses on NVFP4 inference. Lower-precision formats have increased throughput, but FP4 leaves fewer opportunities for further gains through precision reduction.

Sparsity is another potential source of throughput improvement. Nvidia’s specifications often use sparse TFLOPS, although the expected 2x benefit from sparsity has not consistently appeared in practice. Rubin introduces a sparsity scheme different from the 2:4 structured sparsity used in Hopper and Ampere and the 4:8 pairwise structured sparsity used in Blackwell.

Limitations of Specialized Serving

Hardware-specialized disaggregated serving introduces flexibility challenges. The optimal prefill-to-decode ratio depends on factors such as model architecture, service-level requirements, and network bandwidth. Workloads can also change over time.

The VR NVL144 CPX has a fixed number and ratio of Rubin and Rubin CPX chips. That makes it less adaptable than a deployment composed of separate VR NVL144 and VR CPX racks, where operators can adjust prefill and decode capacity independently.

Implications for Competing Accelerators

Google TPU

Google’s TPU systems use a 3D Torus scale-up network that supports pods of up to 9,216 TPUs. This large world size enables a wide range of parallelism strategies and provides comparatively low scale-up networking cost per accelerator.

A prefill-focused TPU could help Google maintain performance per dollar for internal workloads and potentially support later external deployments. Google’s topology may also produce higher performance than some Nvidia configurations for particular models and system designs.

AWS Trainium3 Max NVL72 and Meta MTIAv4 SUE72

AWS Trainium3 Max NVL72 and Meta MTIAv4 SUE72 are examples of systems that combine internal demand with rack-scale designs modeled on the NVL72 form factor. Their internal workloads could support the development of prefill-focused accelerators.

A prefill chip paired with Trainium3 Teton-3 Max NVL72 could use Anthropic’s demand for codesign and deployment. AWS would face integration challenges because the tightly packed 1U compute tray has little space for custom EFA NICs alongside four large Rubin GPU packages and eight CPX GPU packages.

One possible approach would be to place EFA NICs in a separate sidecar rack and connect them through external PCIe AEC cables. Without the CX-9’s integrated PCIe switch, Astera Labs dedicated PCIe switches would be needed to connect the Vera CPU, local NVMe storage, Rubin CPX GPUs, and external EFA NICs.

Meta could use its internal inference workloads to support MTIAv4 SUE72 development. OpenAI’s chip project with Broadcom could also compete through codesign with frontier models and internal demand. MTIAv3 is less suited to this category because its 16-GPU world size is smaller.

AMD MI400 and MI500

AMD’s MI400 72-GPU rack-scale system had been positioned as a way to approach Nvidia’s rack-level performance and cost characteristics. Earlier analysis estimated that MI400 could provide lower total cost of ownership per effective FP4 FLOP than a VR200 NVL144 system, while offering 19.8TB/s of memory bandwidth compared with the initially advertised 13.0TB/s for VR200 NVL144.

Nvidia has since raised the stated R200 memory bandwidth to approximately 20.5TB/s per logical GPU by using faster HBM4 speed bins. This brings VR200 NVL144 in line with MI400’s bandwidth while requiring fewer HBM sites.

If MI400’s effective dense FP4 throughput is equal to or lower than VR200 NVL144, AMD could reach the market with a system similar to Nvidia’s VR200 NVL144 after Nvidia had already introduced the CPX-based configuration. The VR200 NVL144 CPX would then have an advantage for long-context workloads because of its specialized prefill hardware.

AMD would need to develop a prefill-focused accelerator in addition to its rack-scale systems and software stack. Unlike Google, AWS, and Meta, AMD has fewer internal workloads that could provide a demand backstop for another chip-development program.

Potential for a Decode-Focused Accelerator

Nvidia has introduced a prefill-focused chip while continuing to use the R200 for decode. A complementary decode-focused accelerator could take the opposite approach: relatively modest compute capacity combined with substantial memory bandwidth.

Such a design could retain the R200’s memory and off-package I/O capabilities while reducing the main compute die. The I/O chiplets could remain large enough to support the same memory and external connections, while the compute die could use fewer compute units and maintain the required HBM edge interfaces.

Further savings could come from higher parametric yields, fewer active streaming multiprocessors, and lower power requirements. These changes would reduce power-delivery and thermal-management costs. This would reverse the CPX strategy: the decode-focused design would preserve much of the HBM while reducing other parts of the system.

The Rubin CPX therefore represents a shift from uniform accelerator deployments toward systems that assign prefill and decode to different hardware. Its compute-heavy, GDDR7-based design targets the resource imbalance in prefill, while R200-class accelerators retain the memory bandwidth needed for decode.