NVIDIA Cosmos-H-Dreams: Bringing Real-Time Generative Simulation to Surgical Robotics
NVIDIA Cosmos-H-Dreams: Bringing Real-Time Generative Simulation to Surgical Robotics
NVIDIA Cosmos-H-Dreams：将实时生成式仿真引入手术机器人领域
Surgical robotics is moving quickly from teleoperation toward increasingly capable vision-language-action policies. But evaluating and training these systems remains difficult. Physical robotic platforms are expensive to operate, experiments are slow to reproduce, and failures can damage instruments or biological material. Conventional simulators provide a safer alternative, but surgical scenes are exceptionally difficult to model: deformable tissue, fine instrument interactions, specular surfaces, sutures, needles, smoke, and occlusions all matter.
手术机器人技术正迅速从远程操作转向具备更强能力的“视觉-语言-动作”策略。然而，评估和训练这些系统仍然困难重重。物理机器人平台运行成本高昂，实验难以复现，且故障可能损坏器械或生物组织。传统的仿真器提供了一种更安全的替代方案，但手术场景极难建模：可变形组织、精细的器械交互、镜面反射表面、缝合线、缝针、烟雾以及遮挡等因素都至关重要。
World foundation models offer a different path. Instead of manually authoring every object and physical interaction, they learn visual dynamics directly from synchronized video and robot kinematics. NVIDIA’s Cosmos-H-Surgical-Simulator demonstrated this approach by generating future surgical video from an initial scene and a sequence of robot actions. It enabled faster-than-physical evaluation and synthetic data generation across the Open-H-Embodiment ecosystem.
世界基础模型提供了一条不同的路径。它们无需手动构建每一个物体和物理交互，而是直接从同步的视频和机器人运动学数据中学习视觉动力学。NVIDIA 的 Cosmos-H-Surgical-Simulator 通过根据初始场景和一系列机器人动作生成未来的手术视频，展示了这一方法。它实现了比物理实验更快的评估速度，并在 Open-H-Embodiment 生态系统中实现了合成数据生成。
Today, we are introducing the next step: Cosmos-H-Dreams, a real-time, action-conditioned generative simulator for surgical robotics. Cosmos-H-Dreams distills the capabilities of Cosmos-H-Surgical-Simulator into a causal, few-step student model and serves it through FlashDreams, NVIDIA’s accelerated streaming-inference library. Running on a single NVIDIA RTX PRO 6000 GPU, the result is an interactive environment that a person or a learned policy can control in a closed loop.
今天，我们推出了下一步成果：Cosmos-H-Dreams，这是一款用于手术机器人的实时、动作条件生成式仿真器。Cosmos-H-Dreams 将 Cosmos-H-Surgical-Simulator 的能力提炼为一个因果性的、少步（few-step）学生模型，并通过 NVIDIA 的加速流式推理库 FlashDreams 进行服务。该模型可在单张 NVIDIA RTX PRO 6000 GPU 上运行，提供了一个交互式环境，供人类或学习到的策略进行闭环控制。
1. From Surgical World Model to Interactive Simulator
1. 从手术世界模型到交互式仿真器
Cosmos-H-Surgical-Simulator is an action-conditioned world foundation model built on NVIDIA Cosmos-Predict2.5-2B and post-trained on the Open-H-Embodiment dataset. Given a surgical context frame and a future robot trajectory, it generates video showing the likely visual consequences of those actions. This makes it useful for offline policy evaluation and synthetic data generation. A recorded or policy-generated trajectory can be sent to the model, the corresponding rollout can be generated, and the result can be inspected or scored without repeatedly executing the motion on a physical robot.
Cosmos-H-Surgical-Simulator 是一个基于 NVIDIA Cosmos-Predict2.5-2B 构建并经 Open-H-Embodiment 数据集后训练的动作条件世界基础模型。给定手术上下文帧和未来的机器人轨迹，它能生成视频，展示这些动作可能产生的视觉后果。这使其在离线策略评估和合成数据生成方面非常有用。记录的或由策略生成的轨迹可以发送给模型，生成相应的推演结果，从而无需在物理机器人上重复执行动作即可对结果进行检查或评分。
Cosmos-H-Dreams moves the model into the real-time regime. Starting from the multi-embodiment surgical priors learned by Cosmos-H-Surgical-Simulator, we specialize the model for da Vinci Research Kit (dVRK) tabletop suturing and distill it into a causal student that generates the scene autoregressively. The released model receives an initial RGB frame and a live stream of robot kinematics, then produces the next chunk of frames before continuing with the following action block. We have also demonstrated the versatility of Cosmos-H-Dreams by collaborating with CMR Surgical and Cambridge Consultants to integrate it with the Versius surgeon controller, enabling real-time operation on the Versius platform.
Cosmos-H-Dreams 将该模型带入了实时领域。从 Cosmos-H-Surgical-Simulator 学习到的多具身手术先验知识出发，我们将模型专门针对达芬奇研究套件 (dVRK) 的桌面缝合任务进行了优化，并将其提炼为一个自回归生成场景的因果学生模型。发布的模型接收初始 RGB 帧和实时的机器人运动学数据流，然后生成下一组帧，并继续处理后续的动作块。我们还通过与 CMR Surgical 和 Cambridge Consultants 合作，将其与 Versius 外科医生控制器集成，实现了在 Versius 平台上的实时操作，展示了 Cosmos-H-Dreams 的多功能性。
2. Distilling Cosmos-H-Surgical-Simulator for Real Time
2. 为实时性提炼 Cosmos-H-Surgical-Simulator
The key challenge is preserving useful surgical dynamics while reducing the cost of generation. Cosmos-H-Dreams uses a teacher-to-student training pipeline designed for long, autoregressive rollouts.
核心挑战在于在降低生成成本的同时保留有效的手术动力学。Cosmos-H-Dreams 使用了一种专为长序列、自回归推演设计的“教师-学生”训练流程。
2.1. A Surgical Teacher
2.1. 手术教师模型
The bidirectional teacher begins from the Cosmos-H-Surgical-Simulator Open-H checkpoint, which uses a unified 44-dimensional action representation. For the released dVRK tabletop model, the dual-arm dVRK action content, consisting of relative end-effector translation, rotation, and gripper state, is mapped into this common representation. The teacher is then fine-tuned on the JHU dVRK tabletop mixture, including successful demonstrations as well as failure and out-of-distribution episodes such as needle drops, missed throws, and unsuccessful knot ties. These failures are important: a simulator intended to evaluate policies must reproduce the consequences of poor actions, not only ideal demonstrations. To improve stability during long rollouts, the training process progressively increases the teacher’s temporal horizon. We start training on a 12-frame horizon and progressively increase this number until 72 frames. At each horizon bump, we initialize the warmed-up model with pretrained weights.
双向教师模型始于 Cosmos-H-Surgical-Simulator 的 Open-H 检查点，该检查点使用统一的 44 维动作表示。对于发布的 dVRK 桌面模型，双臂 dVRK 的动作内容（包括末端执行器的相对平移、旋转和夹爪状态）被映射到这一通用表示中。随后，教师模型在 JHU dVRK 桌面混合数据集上进行微调，其中包括成功的演示以及失败和分布外的情景，如掉针、投针失败和打结失败。这些失败案例至关重要：旨在评估策略的仿真器必须能够重现错误动作的后果，而不仅仅是理想的演示。为了提高长序列推演的稳定性，训练过程逐步增加了教师的时间跨度。我们从 12 帧的跨度开始训练，并逐步增加到 72 帧。在每次跨度提升时，我们都会使用预训练权重初始化已预热的模型。
2.2. Causal Warmup
2.2. 因果预热
The teacher’s denoising trajectories are first precomputed and cached. A causal student is initialized from the teacher and trained to imitate these cached trajectories. This warmup stage teaches the student to operate with causal attention and a streaming key/value cache before it begins learning from its own generated history.
教师模型的去噪轨迹首先被预计算并缓存。因果学生模型从教师模型初始化，并被训练以模仿这些缓存的轨迹。这一预热阶段教会学生在开始从自身生成的历史中学习之前，如何利用因果注意力机制和流式键/值缓存进行操作。
2.3. Self-Forcing Distillation
2.3. 自强制提炼
Autoregressive models face a familiar problem: during training they may see clean, ground-truth context, while during deployment they must condition on their own imperfect outputs. Small errors can therefore compound over time. Cosmos-H-Dreams addresses this mismatch with self-forcing distillation. During training, the student rolls forward using its own generated context. Distribution-matching supervision from the frozen teacher then guides those self-generated rollouts toward realistic surgical video. This prepares the student for the same conditions it will encounter during interactive inference. The resulting model supports few-step diffusion, with as few as two denoising steps per latent frame, rather than the many-step process used by the full teacher. It combines the teacher’s surgical priors with the causal structure needed for streaming.
自回归模型面临一个常见问题：在训练期间，它们可能看到干净的真实上下文，而在部署期间，它们必须基于自身不完美的输出进行条件化。因此，微小的误差会随时间累积。Cosmos-H-Dreams 通过自强制提炼解决了这种不匹配。在训练过程中，学生模型使用其自身生成的上下文向前推演。来自冻结教师模型的分布匹配监督随后引导这些自生成的推演向真实的手术视频靠拢。这使学生模型能够适应其在交互式推理过程中将遇到的相同条件。最终的模型支持少步扩散，每个潜在帧仅需低至两步的去噪过程，而非完整教师模型所使用的多步过程。它将教师的手术先验知识与流式传输所需的因果结构结合在了一起。
3. FlashDreams: The Real-Time Inference Engine
3. FlashDreams：实时推理引擎
Model distillation is only part of the real-time story. Cosmos-H-Dreams is served through FlashDreams, an accelerated inference library for autoregressive world and video models. FlashDreams turns the distilled student into a low-latency streaming system.
模型提炼只是实现实时性的一部分。Cosmos-H-Dreams 通过 FlashDreams 提供服务，这是一个针对自回归世界模型和视频模型的加速推理库。FlashDreams 将提炼后的学生模型转化为一个低延迟的流式系统。