NVIDIA Cosmos3-H-Dreams generates surgical video at 43.2 fps

NVIDIA released Cosmos3-H-Dreams on October 20, 2026, a world model that generates surgical video in real time from a robot's instrument commands.

NVIDIA announced the model in a post on its Hugging Face blog and put the checkpoint and companion code up the same day on Hugging Face and GitHub. Cosmos3-H-Dreams is fine-tuned on clinical cholecystectomy recordings, that is gallbladder removal, from CMR Surgical's Versius system, the model card states. Give it one endoscopic camera frame and a stream of commands for both robot arms and it generates the endoscopic video that should follow, in four-frame blocks, so the scene responds while an operator is still moving the controls.

The first Cosmos-H-Dreams, built on Cosmos-Predict2.5, simulated dVRK tabletop suturing exercises. The new release moves the same approach inside the body and keeps a narrow scope: training used only windows in which exactly two instruments were visible and engaged, a left Cadiere grasper and a right monopolar hook.

The card lists 15.17 billion parameters and a default rollout of 72 frames at 10 Hz, which is 7.2 seconds, at 832x480. Two checkpoints ship. NVIDIA measures the four-step model at 30.0 frames per second and the separately trained two-step model at 43.2 frames per second, both on an RTX PRO 6000 Blackwell Workstation Edition. The repository asks for more than 32GB of VRAM and drives the simulation from a WebXR headset browser with two tracked controllers, or replays recorded actions into an MP4.

NVIDIA describes Cosmos3-H-Dreams as, to its knowledge, the first real-time generative surgical simulator trained on clinical cholecystectomy recordings that supports live kinematic control. There is no physics engine in the loop. Rather than model geometry and tissue deformation, the model learned from synchronized video and robot telemetry how the endoscope image changes when the instruments move.

NVIDIA also states what it has not shown. Visual realism is a starting point for validation and not proof of biomechanical accuracy, the announcement says, and the company calls Cosmos3-H-Dreams a research simulator, not a validated system for patient care. The held-out evaluation covers about 10.6 minutes drawn from 10 recordings, and the published evaluation is NVIDIA's own. Data Phoenix notes that the published adaptation guide is a reference recipe rather than an executable pipeline, and that the loader does not ship the model's 28-dimensional CMR control adapter.

For people who make video rather than robots, 43.2 frames per second at 832x480 from a 15-billion-parameter model on one workstation card puts generative video inside a live control loop. If that holds on other hardware, interactive generated video becomes practical to drive live.

The checkpoints are on Hugging Face under the NVIDIA Open Model License, which the card says allows commercial and non-commercial use, and the code is on GitHub under Apache-2.0.