NVIDIA’s Cosmos3-H-Dreams generates action-responsive cholecystectomy scenes from clinical recordings, offering researchers a real-time environment for studying surgical interaction.
Hugging Face published details on October 20, 2026, about Cosmos3-H-Dreams, an NVIDIA generative simulator designed to produce video of gallbladder-removal procedures in response to surgical instrument movements. The model learns from synchronized clinical video and robot-kinematics recordings captured on the CMR Surgical Versius platform. Its authors describe it as, to their knowledge, the first real-time generative surgical simulator trained on clinical cholecystectomy recordings that supports live kinematic control.
The system builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on the Open-H-Embodiment dataset. For the surgical adaptation, the team selected footage featuring two visible instruments: a left Cadiere grasper and a right monopolar hook. Its control input has 28 dimensions, covering both arms’ poses and gripper states, energy settings, and thumbstick controls. Training video is 832 by 480 pixels, with actions aligned at 10 Hz; that training rate is not the model’s runtime video-generation speed.
NVIDIA provides two model checkpoints, using four or two denoising steps per generated block. The two-step version is separately trained and trades some visual quality for faster inference. Both are designed for streaming generation, with FlashDreams accelerating inference. The team says the system runs interactively on one NVIDIA RTX PRO 6000 GPU. Demonstrations use different recorded instrument actions to create different continuations from the same starting frame.
The release is aimed at research, not patient care. The article cautions that visual plausibility does not establish biomechanical accuracy, and describes the simulator as unvalidated for clinical use. Suggested next steps include broader procedure coverage, surgeon-led evaluation, support for more instruments, and combining generative video with numerical models of geometry and tissue mechanics. Researchers can also adapt the approach to other datasets and compatible action representations, provided their controllers use matching action conventions and normalization.