NVIDIA Releases Four Cosmos 3 Open Models for Interactive World Simulation
NVIDIA Releases Four Cosmos 3 Open Models for Interactive World Simulation Published November 5, 2026 NVIDIA has added four open weight checkpoints to its Cosmos 3 world model f...
By Hardware Team
NVIDIA Releases Four Cosmos 3 Open Models for Interactive World Simulation
Published November 5, 2026
NVIDIA has added four open-weight checkpoints to its Cosmos 3 world-model family. The models generate interactive simulations and synchronized, multi-sensor driving scenes for robotics and autonomous-vehicle research.
All four checkpoints are post-trained versions of Cosmos 3 Nano with 16B trainable parameters. They use OpenMDW1.1, which permits commercial and non-commercial use under its terms.
The four checkpoints
- Cosmos3-Nano-Transfer-Auto: Generates multi-view driving data under varied weather, lighting, and time-of-day conditions.
- Cosmos3-Nano-Sim-Auto: Streams synchronized camera and LiDAR observations for driving simulation.
- Cosmos3-Nano-Sim-Transfer: Converts simulator controls and text prompts into photorealistic video.
- Cosmos3-Nano-Sim-Bimanual: Predicts video from robot or camera actions.
Why these models are useful
Robotics requires simulation environments that respond to actions across changing scenes and viewpoints. Sim-Bimanual predicts visual consequences from robot actions, while Sim-Transfer converts physics-simulator controls into generated observations.
Autonomous-driving systems require visual appearance and geometry to remain consistent. Synchronized camera and LiDAR generation connects image observations with three-dimensional sensing. World Scenario Map (WSM) controls allow researchers to vary road layouts, scene structure, traffic motion, and traffic-signal states while studying how driving policies respond.
Streaming generation allows each observation to influence the next action. This links video prediction with ongoing interaction and provides a way to study how actions and observations develop together.
The open weights and post-training recipes provide a starting point for adapting the models, comparing approaches, and contributing improvements. Evaluation remains necessary to determine where the models support policy learning and testing accurately, and where their limitations affect results.
Building autonomous-vehicle worlds with Cosmos 3 Nano
Transfer-Auto: driving scenarios to multi-sensor data
Cosmos3-Nano-Transfer-Auto is a bidirectional model for offline synthetic driving-data generation. It can produce synchronized video for up to 11 camera views, including fisheye cameras, and supports joint RGB and LiDAR generation.
Per-camera World Scenario Map controls define scene structure and traffic movement, while text prompts guide visual appearance. Environmental conditions can be varied to expand training and evaluation coverage. Initial RGB images can also be supplied to anchor the scene’s appearance.
Joint LiDAR generation uses aligned HD-map range-map controls. Transfer-Auto can also be adapted to a specific sensor rig or driving domain. Post-training on custom data can account for a project’s camera arrangement, LiDAR configuration, and target environment. The adapted model can then generate offline data or undergo causal training and few-step distillation for faster streaming simulation, which is the development path used for Cosmos3-Nano-Sim-Auto.
Sim-Auto: driving controls to streaming observations
Cosmos3-Nano-Sim-Auto is the causally distilled streaming version of Transfer-Auto. It generates successive short segments of synchronized camera video and LiDAR in a few denoising steps. Generated history and updated driving controls support streaming and long interactive rollouts.
The model accepts per-camera captions and synchronized WSM controls, with optional initial RGB images. LiDAR generation uses HD-map range controls aligned with the camera controls. Each new segment is generated from the rolling history.
Closed-loop interaction requires a simulation runtime and a WSM renderer that update the controls from driving inputs. For a new sensor rig or domain, Transfer-Auto is adapted first and then distilled into Sim-Auto.
Sim-Transfer: simulator controls to RGB
A physics simulator calculates motion, contacts, and physical state. Sim-Transfer uses spatial controls from that simulator, together with a text prompt, to generate color observations for a policy.
Action -> simulator update -> spatial controls -> generated RGB -> next policy action
Supported control types include:
- Depth: Relative distance from the camera.
- Segmentation: Regions and object boundaries.
- Edges: Scene contours.
- Blur: Coarse appearance and layout.
Text guides appearance, while physics and ground-truth state remain in the simulator. The system can, for example, convert depth or segmentation controls into video.
Sim-Bimanual: actions to video
Cosmos3-Nano-Sim-Bimanual models how an environment changes in response to actions from a bimanual robot. It supports interactive two-arm simulation for YAM and AgiBot embodiments, controlled by a policy or human-operated joysticks.
The process starts with an image. The robot or camera is moved, and the model generates the next observations. A single control sequence can combine viewpoint changes with manipulation.
The mixed-action example uses 59 control values, padded to 64 during training. Its schema specifies coordinate frames, rotations, gripper calibration, validity masks, and control domains. These conventions must be followed so that each value represents the intended motion.
Timing must also remain aligned: T transitions require T+1 observations. Training transforms raw captures into the model’s action representation, while released inference actions are already normalized.
Sim-Transfer generates visuals from the updated state of a physics simulator. Sim-Bimanual instead generates the next observation directly from the scene and robot actions.
The published setup requires Linux, an NVIDIA GPU, Docker, and NVIDIA Container Toolkit. The vLLM-Omni configuration was tested on one GB200, which is not a minimum-memory specification. The server downloads the model weights, while clients retrieve sample inputs.
How streaming generation works
Cosmos 3 uses a Mixture-of-Transformers backbone that connects separate text-processing and visual-generation networks through attention. The generator predicts each short segment from the prompt, available controls, and generated history.
Controls + history -> generate chunk -> decode images -> update history
Denoising converts noisy, compressed video representations into a segment. Temporal-causal attention can read available history and controls while blocking future target observations. A key/value cache preserves attention information for the next prediction.
The models operate on compressed video representations called latents. After the initial image, each latent time step represents four RGB frames. Transfer predicts one step per chunk, while the optimized Bimanual configuration described in the report predicts two. Smaller chunks allow new controls to affect generation more frequently.
The four-stage training recipe
- Bidirectional adaptation: The model learns the target domain and control-to-video relationships from complete recorded clips.
- Teacher forcing: Each chunk is predicted from earlier recorded observations and available controls, while the clean answer and future targets are masked.
- Consistency distillation: A frozen teacher trains a student that generates each chunk in four denoising steps.
- Self-forcing: The model is trained on history generated by the student. A frozen teacher and an auxiliary score model provide the distribution-matching signal.
The final stage addresses compounding errors. A slightly incorrect object can become context for later frames, so training on generated history exposes the model to the conditions it will encounter during rollout. Distillation and caching reduce the cost of generating each observation.
Adapting the models with custom data
The post-training recipes start from Cosmos3-Nano or compatible base weights and cover all four training stages. Full training checkpoints should be retained for resuming work. Exported inference weights do not include optimizer state or other training information.
- Transfer-Auto: Post-trains on driving data to adapt to a sensor rig and domain, then generates offline synthetic data.
- Sim-Auto: Applies causal training and distillation to an adapted Transfer-Auto model for fast streaming simulation.
- Sim-Transfer: Post-trains the checkpoint for a target environment with local data preparation, stage configurations, and resume support.
- Sim-Bimanual: Post-trains for a selected embodiment with data validation, stage configurations, checkpoint handoffs, and export.
A typical workflow provides the training procedure and dataset location to a coding agent, checks the environment, validates a sample, and tests training and restart behavior before a longer run. Quality should be evaluated on held-out episodes. A successful short run verifies execution, not convergence.