NVIDIA Cosmos3-H-Dreams Brings Clinical Surgery into Generative Simulation
NVIDIA Cosmos3 H Dreams Brings Clinical Surgery into Generative Simulation NVIDIA has introduced Cosmos3 H Dreams, an interactive generative simulator designed for the visual an...
By Hardware Team
NVIDIA Cosmos3-H-Dreams Brings Clinical Surgery into Generative Simulation
NVIDIA has introduced Cosmos3-H-Dreams, an interactive generative simulator designed for the visual and procedural complexity of clinical surgery. The system can generate surgical scenes that respond to instrument movements, such as moving a Cadiere grasper, grasping the gallbladder, and lifting it within a simulated environment.
The first Cosmos-H-Dreams release, built on Cosmos-Predict2.5, applied interactive generative simulation to tabletop suturing with the dVRK platform. That setting provided a controlled environment for surgical robotics. Instead of explicitly modeling scene geometry, tissue deformation, and physical interactions, the world foundation model learned visual dynamics from synchronized video and robot kinematics.
Cosmos3-H-Dreams extends this approach to clinical procedures recorded on the CMR Surgical Versius platform. The project describes itself as the first real-time generative surgical simulator trained on clinical cholecystectomy recordings and designed to support live kinematic control. Based on Cosmos 3 and accelerated with FlashDreams, NVIDIA’s streaming-inference library, it runs interactively on a single NVIDIA RTX PRO 6000 GPU.
1. From Tabletop Tasks to Clinical Scenes
In a tabletop environment, instruments operate against a relatively simple background. Inside the human body, small movements can affect the entire scene. Tissue stretches under traction, reflections change, anatomical structures become occluded, and smoke can reduce visibility. A generative simulator must reproduce these effects while preserving the appearance of anatomy and instruments across successive video blocks.
Small errors may accumulate during generation, gradually altering instrument geometry or anatomical structure. Temporal consistency is therefore essential. Visual realism is only a starting point for validation and does not establish biomechanical accuracy.
The generated scene must also respond correctly to controls. Translation, rotation, and gripper commands should produce coherent anatomical consequences. A scene that looks plausible but ignores the supplied actions cannot function as an interactive environment.
Cosmos3-H-Dreams addresses this requirement with a model specialized for clinical cholecystectomy and distilled for real-time generation. It combines action-conditioned interaction with complex anatomy, tissue motion, and occlusion while maintaining the low latency needed for a live control loop.
2. What Is New in Cosmos3-H-Dreams?
2.1 A Cosmos 3 Foundation and Expanded Surgical Controls
Cosmos3-H-Dreams is built on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on the Open-H-Embodiment dataset. A bidirectional teacher model is then specialized using curated cholecystectomy video and kinematic data.
The training data focuses on scenes containing two visible, engaged instruments: a left Cadiere grasper and a right monopolar hook. Camera commands are limited to emphasize interaction driven by the instruments.
The model uses 28 action dimensions, including pose and gripper information for both arms, energy controls, thumbstick axes, and thumbstick buttons. Training uses 832 x 480 video with actions aligned at 10 Hz. This sampling rate is separate from the system’s runtime generation throughput.
2.2 From Teacher to Four-Step and Two-Step Students
Cosmos3-H-Dreams uses a teacher-to-student training pipeline intended for long autoregressive rollouts, together with FlashDreams for accelerated streaming inference.
The bidirectional teacher is first adapted using 24 future frames, then extended to a 72-frame horizon. At 10 Hz, these sequences represent 2.4 and 7.2 seconds, respectively, giving the teacher longer windows in which to learn surgical dynamics.
Distillation then converts the bidirectional teacher into a causal student that generates video in short blocks as new actions arrive. Causal teacher forcing prepares the student for sequential generation from recorded context. Self-Forcing trains the model on its own generated context, using guidance from the teacher and a learned critic. This process is intended to reduce the accumulation of errors during interactive rollouts.
The first released student uses four denoising steps for each generated block. A second checkpoint is trained separately with a two-step schedule. It is not simply the four-step model executed with fewer sampling steps.
Using fewer denoising steps reduces inference time but also lowers visual quality. The two checkpoints allow developers to examine the trade-off between visual quality and inference cost.
2.3 Streaming with FlashDreams
FlashDreams provides the accelerated streaming runtime. Long rollouts also require the system to manage the model’s attention history. The Cosmos3-H-Dreams integration uses an adapted FadeMem module to consolidate the growing key/value cache during extended generation.
These components serve different purposes. Self-Forcing addresses how the model learns from generated history, while cache consolidation addresses how that history is stored during streaming inference.
3. Exploring Unseen Trajectories
Clinical procedures can involve complications or movements that are not present in the training data. A simulator must therefore generalize beyond its demonstrations and produce coherent frames for previously unseen actions.
Cosmos3-H-Dreams can generate different cholecystectomy scenes conditioned on instrument actions recorded with a VR headset. Demonstrations begin from the same initial frame but use different action sequences, producing distinct continuations of the scene.
These rollouts provide a way to inspect whether instruments follow the supplied controls, whether tissue responds coherently to interaction, and whether anatomical and instrument appearances remain consistent over time.
4. Adapting the System to Other Data and Controllers
Cosmos3-H-Dreams provides four-step and two-step checkpoints for cholecystectomy. The system can also be adapted to other datasets and embodiments that contain synchronized video and compatible action representations.
Training a real-time student model on a new dataset requires teacher fine-tuning and Self-Forcing distillation. The associated training recipe describes data preparation, public teacher-training tools, and the distillation stages. Controller integrations must follow the same action conventions and normalization used during training.
5. Research Directions
The improved visual realism creates several areas for further research:
- Model the complete procedure: Expand coverage and temporal consistency so all phases of cholecystectomy can be explored systematically, with defined task boundaries and success criteria.
- Combine generative and numerical simulation: Pair learned visual realism with explicit geometry, contact modeling, and tissue mechanics. Numerical models and measured data could help constrain and validate deformation, making behavior more inspectable.
- Evaluate the system with surgeons: Assess anatomical accuracy, instrument behavior, and tissue response, and convert expert feedback into reproducible benchmarks.
- Support additional instruments: Extend the system beyond two controlled tools to represent richer coordination, occlusion, and interaction.
- Develop autonomous surgical policies: Connect world action models to Cosmos3-H-Dreams in a closed loop, allowing policies to learn defined cholecystectomy subtasks.
These remain research directions rather than established clinical capabilities. Cosmos3-H-Dreams is a research simulator, not a validated system for patient care. Its purpose is to support the study of surgical interaction in a more realistic simulated environment.
6. Models, Code, and Data
The resources associated with Cosmos3-H-Dreams include: