NVIDIA Cosmos-H-Dreams Brings Real-Time Simulation to Surgical Robots

NVIDIA has introduced Cosmos-H-Dreams, a real-time, action-conditioned generative world model for surgical robotics. It distills the visual dynamics learned by Cosmos-H-Surgical-Simulator into a causal, few-step student model via self-forcing distillation, then serves it through FlashDreams, achieving roughly 160 frames per second on a single NVIDIA RTX PRO 6000 GPU.

This approach sidesteps the difficulty of manually modeling deformable tissue, fine instrument interactions, specular surfaces, and occlusions by learning directly from synchronized video and robot kinematics. The teacher model is fine-tuned on dVRK tabletop suturing data that includes failure cases, so the resulting simulator reproduces the consequences of poor actions rather than only idealized demonstrations.

Cosmos-H-Dreams supports multiple human-machine interfaces, including a browser client with keyboard controls, a Meta Quest client via WebXR, and closed-loop connection to learned surgical policies. NVIDIA demonstrated real-time operation on CMR Surgical's Versius platform through a collaboration with Cambridge Consultants, and provides a step-by-step guide for teams wanting to train real-time student models on their own robotic embodiments.