New NVIDIA Cosmos 3 Open Models for Interactive World Simulation

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

New NVIDIA Cosmos 3 Open Models for Interactive World Simulation

NVIDIA Cosmos 世界模型开发平台为其 Cosmos 3 模型系列新增了四个开放权重检查点，用于生成交互式世界模拟和多传感器多视角驾驶场景：

• Cosmos3-Nano-Transfer-Auto: Generate multi-view driving data with varied weather, lighting, and time of day.
• Cosmos3-Nano-Sim-Auto: 流式传输同步的摄像头和激光雷达观测数据，用于驾驶模拟。
• Cosmos3-Nano-Sim-Transfer: Turn simulator controls and a text prompt into photorealistic video.
• Cosmos3-Nano-Sim-Bimanual: 根据机器人或摄像头的动作预测视频。

All four models are post-trained examples of Cosmos 3 Nano and have 16B trainable parameters. The models use OpenMDW1.1, permitting commercial and non-commercial use under its terms.

这四个模型均为 Cosmos 3 Nano 的后训练示例，拥有 160 亿个可训练参数。这些模型使用 OpenMDW1.1 协议，允许在条款范围内进行商业和非商业用途。

Why do you need these checkpoints?

机器人技术需要能够跨越不断变化的场景和视角来回答问题的环境。这些检查点提供了构建它们的互补方式：Sim-Bimanual 可以根据动作预测视觉后果，而 Sim-Transfer 则将物理模拟器的控制指令转化为生成的观测结果。

For autonomous driving, appearance and geometry must agree. Synchronized camera and LiDAR generation connects visual observations with 3D sensing. Scene and traffic-signal controls let researchers vary road layouts and signal states, then study how a driving policy responds.

对于自动驾驶而言，外观和几何结构必须保持一致。同步的摄像头和激光雷达生成将视觉观测与 3D 感知连接起来。场景和交通信号控制允许研究人员改变道路布局和信号状态，进而研究驾驶策略如何做出响应。

Each observation can shape the next action. Streaming generation connects video prediction to ongoing interaction, giving world-model researchers a way to explore how actions and observations unfold together.

每一次观测都能塑造下一个动作。流式生成将视频预测与持续的交互连接起来，为世界模型研究人员提供了一种探索动作与观测如何共同演变的方法。

An open foundation for others to build on. NVIDIA’s vision is to help researchers and developers create richer, more responsive simulated worlds. Open weights and post-training recipes provide a shared starting point for adapting models, comparing methods, and contributing improvements.

一个供他人构建的开放基础。NVIDIA 的愿景是帮助研究人员和开发人员创建更丰富、响应更灵敏的模拟世界。开放权重和后训练配方为调整模型、比较方法和贡献改进提供了一个共享的起点。

Build Autonomous Vehicle Worlds with Cosmos3-Nano

Cosmos3-Nano-Transfer-Auto 是一个用于离线合成驾驶数据生成的双向模型。它能生成多达 11 个摄像头视角（包括鱼眼镜头）的同步视频，并支持 RGB 与激光雷达的联合生成。

Per-camera World Scenario Map (WSM) controls specify scene structure and traffic motion, while text prompts guide appearance. Developers can vary environmental conditions to expand training and evaluation coverage, with optional initial RGB images to anchor scene appearance.

每个摄像头的世界场景地图 (WSM) 控制项用于指定场景结构和交通运动，而文本提示则引导外观。开发人员可以改变环境条件以扩大训练和评估的覆盖范围，并可选择使用初始 RGB 图像来锚定场景外观。

Joint LiDAR generation additionally uses aligned HD-map range-map controls. Transfer-Auto also serves as the foundation for adapting Cosmos to your own sensor rig and driving domain.

联合激光雷达生成还使用了对齐的高精地图范围图控制。Transfer-Auto 也可作为将 Cosmos 适配到您自己的传感器设备和驾驶领域的基石。

Sim-Auto: driving controls → streaming observations

Cosmos3-Nano-Sim-Auto 是 Transfer-Auto 的因果蒸馏流式变体。它仅需几个去噪步骤即可生成连续的小块同步摄像头视频和激光雷达数据，利用生成的历史记录和更新的驾驶控制指令来支持快速流式传输和长期的交互式演练。

Provide per-camera captions and synchronized WSM controls, optionally adding initial RGB images. For LiDAR generation, include HD-Map range controls aligned with the camera controls.

提供每个摄像头的说明文字和同步的 WSM 控制项，并可选择添加初始 RGB 图像。对于激光雷达生成，需包含与摄像头控制对齐的高精地图范围控制。

The distilled model generates successive chunks using its rolling history. Closed-loop interaction requires a simulation runtime and WSM renderer to update those controls from driving inputs.

该蒸馏模型利用其滚动历史记录生成连续的数据块。闭环交互需要模拟运行时环境和 WSM 渲染器，以便根据驾驶输入更新这些控制指令。

Sim-Transfer: simulator controls → RGB

您的模拟器负责计算运动、接触和物理状态。Sim-Transfer 使用其空间控制指令和文本提示来渲染策略可以使用的颜色观测结果。

Action → simulator update → spatial controls → generated RGB → next policy action. Depth: relative distance from the camera. Segmentation: regions and object boundaries. Edges: scene contours. Blur: coarse appearance and layout.

动作 → 模拟器更新 → 空间控制 → 生成的 RGB → 下一个策略动作。深度：相对于摄像头的距离。分割：区域和物体边界。边缘：场景轮廓。模糊：粗略的外观和布局。

Sim-Bimanual: actions → video

Cosmos3-Nano-Sim-Bimanual 模拟了环境如何响应双臂机器人的动作而演变。它支持针对 YAM 和 AgiBot 实施方案的交互式双臂模拟，由策略或人工操作的操纵杆进行控制。

Start with an image, move the robot or camera, and generate the next observations. One control sequence can switch between viewpoint changes and manipulation.

从一张图像开始，移动机器人或摄像头，并生成后续的观测结果。一个控制序列可以在视角切换和操作之间进行转换。