NVIDIA NVLink Fusion Brings NVHBM to Next-Generation AI Infrastructure

AI factories must support increasingly large models and more complex reasoning workloads. To meet the growing computational demands of AI, hyperscalers and AI-native companies are developing custom AI accelerators, commonly called XPUs. Deploying these accelerators at scale requires high-bandwidth memory (HBM), enough package and silicon area for compute, efficient power delivery, a resilient supply chain, and a rack-scale architecture for integrating XPUs into data center infrastructure.

NVIDIA NVLink Fusion is the connectivity technology and intellectual property that enables hyperscalers and AI-native companies to deploy custom XPUs and CPUs within the NVIDIA AI infrastructure platform. Through the NVIDIA scale-up and scale-out technology stack, ecosystem, and MGX rack-scale architecture, organizations can reduce development and deployment complexity, improve performance, and accelerate the deployment of semi-custom AI factories.

At the package level, NVHBM complements this architecture. NVHBM is a custom HBM base-die technology designed and validated with leading memory vendors. It is intended to provide higher memory bandwidth, improved area efficiency, and lower power consumption. These changes can help custom XPUs support larger models, read KV cache data more quickly, and improve training and large-scale inference.

Why bandwidth, die area, and power matter in accelerator design

Training, inference, and agentic AI workloads increasingly depend on high-throughput access to model weights, KV cache, and activation data. As AI systems expand from individual accelerators to rack-level compute domains, accelerator packages must balance compute logic, power delivery, thermal design, and high-bandwidth memory.

HBM places high-speed memory close to the accelerator, but qualifying memory technology, integrating it into a package, and completing validation can create bottlenecks for custom accelerator programs. Through NVLink Fusion, customers gain access to NVHBM base dies validated with leading memory manufacturers, which can help reduce integration and qualification challenges.

Table 1. NVHBM provides higher memory bandwidth, additional package and silicon area, and lower HBM power usage for AI accelerator programs.

The memory bandwidth bottleneck in modern AI accelerators

AI accelerator performance depends on how consistently compute engines receive data. Higher HBM speeds increase usable memory bandwidth within a given package budget, improving support for bandwidth-intensive phases of training and inference.

NVHBM provides up to 30% more memory bandwidth per stack than standard HBM4e. For memory-bound or partially memory-bound workloads, this can improve accelerator utilization and throughput. During large-model inference, faster transfers between HBM and compute cores can also increase per-user token throughput by keeping the compute engines supplied with data.

While NVHBM increases memory bandwidth within each accelerator, NVLink Fusion connects accelerators across larger domains so workloads can use distributed compute and memory more efficiently.

This scale-up capability is particularly important for routing methods such as expert parallelism (EP) and WideEP. In these configurations, different experts reside on different GPUs and require high-speed synchronization across the rack. NVIDIA NVLink, the scale-up networking fabric for AI factories, transfers activations and hidden states between experts and synchronizes distributed caches across the scale-up fabric. NVHBM helps reduce data starvation by keeping local compute engines supplied with data.

More package area and greater design flexibility

For custom AI silicon, every square millimeter matters. Accelerator designers must determine how much area to assign to matrix engines, vector units, on-chip SRAM, cache hierarchies, control logic, memory interfaces, networks-on-chip, and scale-up connectivity. A custom memory implementation can reduce the design and package overhead associated with HBM access, leaving more area for workload-specific capabilities.

As AI workloads diversify, this additional die area gives hyperscalers more flexibility to optimize XPUs for inference serving, recommendation systems, multimodal pipelines, or internal training. By reducing the area required for the memory interface, NVHBM allows teams to dedicate more of the chip to performance-related functions.

The area savings primarily come from a redesigned physical memory interface, or PHY. Standard HBM uses wider interface connections, which increase the overall package footprint. NVHBM uses a custom base die optimized for efficiency. It reduces I/O area requirements by moving the memory controller into the 3D HBM stack and integrating a custom PHY.

Compared with the JEDEC HBM4e standard, this design reduces PHY and support area by up to 67%. The narrower interface also simplifies interposer routing and provides up to 80% more usable silicon across the layout.

Figure 1. Comparison of die area savings with NVHBM and standard HBM.

Reducing the size of the memory interface allows the central AI compute die to expand into the freed space. This can provide up to a 30% increase in available main-die silicon for compute or other features. The additional area allows XPU designers to add capabilities while maintaining a fixed package footprint.

Power savings for efficient scaling

Power is a major constraint in modern AI infrastructure. HBM power contributes to the accelerator power budget, package thermal design, rack power envelope, and data center cooling requirements. NVHBM enables 15% lower HBM power usage than standard HBM4e, creating additional power and thermal headroom for compute.

Power savings affect multiple levels of the infrastructure. At the XPU level, lower HBM power can improve performance per watt and create room for more compute or higher sustained utilization. At the rack level, it can reduce pressure on power delivery and cooling systems. Across an AI factory, even modest reductions in memory subsystem power can accumulate across thousands of accelerators.

For a 1-gigawatt data center using 2,000W XPUs, the stated power savings can provide enough compute headroom for up to 15,000 additional XPUs.

The benefit is particularly relevant to large-model inference. XPUs repeatedly read model weights and KV-cache data while serving users with low latency and high throughput. Reducing the energy required to move this data can support faster inference on large models, larger batch sizes, and more efficient use of deployed power.

Combining NVLink Fusion with NVHBM at rack scale

NVHBM improves XPU performance and efficiency at the chip level. NVLink Fusion connects those XPUs to the broader NVIDIA AI platform. According to the stated design figures, combining a 30% increase in memory bandwidth, 25% more die area, and 15% HBM power savings results in a 30% overall end-to-end performance increase per XPU.

This connectivity is provided through the NVLink Fusion chiplet, which bridges custom XPUs and the NVLink fabric. It connects all XPUs in a rack into a single scale-up domain. NVLink is in its sixth generation, and the platform also supports connecting XPUs to CPUs through NVLink-C2C.

NVLink Fusion adopters can combine custom XPUs and CPUs with the NVIDIA scale-up and scale-out technology stack and ecosystem. This is intended to reduce development and deployment complexity, increase performance, and accelerate time to market for semi-custom AI factories.

A unified architecture can also simplify data center operations, support flexible reprovisioning of capacity, and allow custom AI XPUs to work with GPUs in heterogeneous computing environments.

The next phase of custom AI silicon

NVLink Fusion provides a common scale-up foundation for GPUs, custom XPUs and CPUs, networking, and rack-level software. NVHBM complements that foundation with higher memory performance, greater compute density, improved HBM power efficiency, and supply resiliency for next-generation accelerators.

Together, these technologies provide partners with a path from custom AI accelerator design to rack-scale deployment and production volume.