Mixx Technologies’ Roadmap to Fully Optical Scale-Up Networks

Modern rack systems such as Nvidia’s NV72 use miles of copper flyover cables to connect six dozen or more GPUs so they can operate as a single system. As Nvidia and other companies pursue larger scale-up fabrics, optical links are becoming unavoidable. However, increasing the scale of these systems also increases the number of fibers emerging from the back of each rack.

That is the first problem Mixx Technologies is trying to solve. Founded in 2023 by Vivek Raghuraman and Rebecca Schaevitz, the photonics startup unveiled its first product earlier this year. The SxC connector can accommodate 64 fibers in a single connector.

“We can put 1,000 fibers within 800mm2,” Raghuraman said.

Raghuraman also argues that larger scale-up computing domains require a different approach to system design. In his view, a rack’s backplane could function as an optical interposer for the silicon inside it. Removing copper’s distance and bandwidth limitations could eventually enable a fully optical compute fabric.

To support that goal, Mixx is developing several related technologies: near-packaged optical engines, copackaged optical engines, optical circuit switches, and a software control plane to manage the network. Raghuraman describes this as a three-part roadmap for optical scale-up networks.

The transition will take time, and that expectation is central to Mixx’s strategy. Many photonics companies have pursued similarly ambitious plans, but Raghuraman has previous experience building optical engines. During his seven years at Intel, his team worked on several silicon photonics advances. He was also directly involved in developing Broadcom’s copackaged optical switches.

Raghuraman says Mixx is focusing on deployable products rather than proof-of-concept systems, with releases timed to market demand.

Nvidia’s expanding optical fabrics

With its Vera Rubin Ultra rack systems, Nvidia is expanding its NVLink Switch fabrics from 72 to 144 GPUs, then to 576 and eventually 1,152 GPUs, with optics supporting communication between racks. Links within each rack will remain copper, while rack-to-rack communication will use optical connections.

Nvidia has not specified the exact implementation. One possibility is the integration of near-packaged or copackaged optics into NVSwitches. Assuming one fiber pair for each SerDes, a Vera Rubin NVL576 system would require approximately 20,736 fiber pairs to connect the eight NVL72 racks from which it is assembled. That amounts to 2,592 fiber pairs per rack.

Dense wavelength-division multiplexing could reduce the number of physical fibers by carrying multiple wavelengths over each fiber. Even with 16 wavelengths per fiber, however, the resulting fiber count would create substantial cleaning and termination requirements.

This is why Raghuraman and his team are starting with the connector. Any company building an optical scale-out network, including AMD, Nvidia, or partners using Broadcom or Marvell XPUs, will need a denser fiber attachment method.

Mixx’s next product is planned as a near-packaged optical, or NPO, engine that could be integrated into a GPU node. Copackaged optics, or CPO, integrated directly into ASIC packages remains the company’s long-term objective, but Raghuraman believes the market is not yet ready for that approach.

NPO is generally considered less risky because an optical engine failure would not necessarily make the associated GPU unusable. In principle, the optical module could be refurbished or replaced. Mixx is targeting 2028 or 2029 for its first NPO module.

NPO could eventually eliminate copper interconnects and simplify the scale-up topology from two tiers into a single, larger tier. Raghuraman expects adoption to begin with near-package optics, which places the optical technology close to the switch or compute device. Once the NPO form factor is proven, the technology could transition to true CPO.

Mixx expects its first CPO product near the end of the decade, around the time SerDes technology may no longer be practical for scale-up networking.

Moving beyond SerDes limitations

SerDes technology is widely used in high-speed networking to transmit data over comparatively long distances, but copper-based SerDes links have reach limits. At 400 Gbps, a signal can travel approximately 1.25 meters over copper before degradation becomes a problem. Doubling the data rate roughly halves the reach.

With CPO, the electrical signal would need to travel only a few millimeters to reach the optical engine. For data center deployments, that makes signal reach far less significant. Moving away from long electrical SerDes links could also change the structure of scale-up networks.

These networks typically connect 72 or more GPUs in an all-to-all fabric. Each GPU can communicate with every other GPU through packet switches such as Nvidia’s NVSwitch or Broadcom’s Tomahawk 6.

The maximum size of a single-layer scale-up fabric is influenced heavily by accelerator radix, meaning the number of ports available on each accelerator. Nvidia’s Blackwell generation has 36 200 Gbps links, with two links connecting each accelerator to each of the 18 NVSwitch ASICs in an NVL72 system.

Raghuraman expects eliminating SerDes links to enable a much higher radix than copper interconnects can support. He says the design is most efficient when it reaches densities comparable to die-to-die interconnects.

A higher radix could reduce or eliminate the need for packet switches in some systems. If packet switching can be avoided, latency can improve. Packet switches would still have a role in larger clusters, but their function could move to a higher layer of the network.

Toward a fully optical scale-up fabric

Systems such as Google’s TPU and Amazon’s Trainium2 pods demonstrate that compute fabrics can be built without packet switches. Historically, however, limited chip-to-chip radix required the use of two-dimensional and three-dimensional torus topologies.

Torus networks work well for training and inference with dense models, but they are less suitable for mixture-of-experts, or MoE, architectures because communication can require more hops than in an all-to-all switched fabric. This limitation contributed to AWS’s decision to move Trainium3 toward a switched fabric resembling Nvidia’s NVL72 design.

With a sufficiently high radix, chips could connect directly to one another while maintaining all-to-all connectivity. Every GPU could then be reached in one hop without packet switches. Mixx is working to reduce the need for packet switches, but its ideal scale-up network would still include switching technology.

Alongside its CPO engines, the company is developing a high-density optical circuit switch, or OCS. Google uses OCS technology in TPU clusters to resize pods and route around failed accelerators. The switches can reconfigure optical paths in a way similar to disconnecting one telephone call and connecting another.

The challenge is that current OCS products generally lack the required density.

“The biggest challenge with OCS today is getting high-radix OCS switches. Most of the technologies today are 16x16, 32x32,” Raghuraman said.

Some companies, including Coherent, now offer OCS products with hundreds of ports, but those systems are also physically large. Mixx may require OCS appliances with thousands of high-density ports. Such systems would be difficult to build without an ultra-dense fiber attachment technology such as SxC.

The network will also require a software control plane to manage optical connections. Reconfiguring an OCS causes connections to drop temporarily. In an all-to-all fabric, these interruptions may be less disruptive because multiple paths can exist between accelerators, but reconfiguration still needs to be scheduled carefully.

Mixx’s fully optical scale-up vision remains years away. Its roadmap is designed to introduce products as market demand develops, rather than producing proof-of-concept systems long before the infrastructure needed to deploy them exists.