Introducing Cosmos 3 Edge

Introducing Cosmos 3 Edge Cosmos 3 Edge 介绍

The real world is vast and to operate in it physical AI systems need to understand how a scene is changing, anticipate what could happen next, and determine how an action will affect the world. Machines operate at the edge in factories, warehouses, hospitals and beyond. To understand and act in these environments, they need models that deliver data center–level performance on memory-constrained systems. 现实世界广阔无垠，物理 AI 系统若要在其中运行，就需要理解场景如何变化、预测接下来可能发生的事情，并确定某个动作将如何影响世界。机器在工厂、仓库、医院等边缘环境中运行。为了在这些环境中理解并采取行动，它们需要能够在内存受限的系统上提供数据中心级性能的模型。

Today we are releasing NVIDIA Cosmos 3 Edge on Hugging Face’s Cosmos 3 repository. It’s a 4-billion-parameter open world model that helps robots and vision AI agents understand their surroundings, reason in real time and generate robot actions on edge devices. Download the model: https://huggingface.co/nvidia/Cosmos3-Edge 今天，我们在 Hugging Face 的 Cosmos 3 存储库中发布了 NVIDIA Cosmos 3 Edge。这是一个拥有 40 亿参数的开放世界模型，旨在帮助机器人和视觉 AI 智能体理解周围环境、进行实时推理，并在边缘设备上生成机器人动作。模型下载地址：https://huggingface.co/nvidia/Cosmos3-Edge

The model delivers memory-efficient, high-throughput inference across NVIDIA edge computers including NVIDIA RTX PRO GPUs, NVIDIA DGX, NVIDIA GeForce RTX™ GPUs, NVIDIA Jetson including the newly announced Jetson T2000 and T3000 modules. Designed as a compact open model that can serve as a small vision language model (VLM) with best-in-class throughput and accuracy with real-time inference. 该模型可在 NVIDIA 边缘计算机上实现内存高效、高吞吐量的推理，包括 NVIDIA RTX PRO GPU、NVIDIA DGX、NVIDIA GeForce RTX™ GPU，以及包括新发布的 Jetson T2000 和 T3000 模块在内的 NVIDIA Jetson 系列。它被设计为一个紧凑的开放模型，可用作小型视觉语言模型 (VLM)，在实时推理中具备同类最佳的吞吐量和准确性。

As a post-trained world action model (WAM), Cosmos 3 Edge operates at robot-control resolution (640×360 observations), delivering real-time reasoning and generating 32 actions per inference on NVIDIA Jetson Thor - while achieving real-time control at 15 Hz. Among similar size (4B parameters) models, Cosmos 3 Edge ranks #1 on VANTAGE-Bench for vision analytics and state-of-the-art for robot policy learning, setting the state of the art for smart infrastructure and robotics. 作为一种经过后训练的世界动作模型 (WAM)，Cosmos 3 Edge 在机器人控制分辨率（640×360 观测值）下运行，在 NVIDIA Jetson Thor 上实现实时推理并每次推理生成 32 个动作，同时以 15 Hz 的频率实现实时控制。在同等规模（40 亿参数）的模型中，Cosmos 3 Edge 在 VANTAGE-Bench 视觉分析任务中排名第一，在机器人策略学习方面处于领先地位，为智能基础设施和机器人技术树立了新的标杆。

What Is World Modeling?

什么是世界建模？

A world model learns how an environment changes over time. It represents objects, motion, spatial relationships, and the effects of actions. Consider a robot reaching for an object. Recognizing the object is only the first step. The robot must also understand where the object is, how its gripper is moving, what may happen when contact occurs, and which action is most likely to complete the task successfully. 世界模型学习环境如何随时间变化。它表示物体、运动、空间关系以及动作的影响。以机器人伸手抓取物体为例，识别物体只是第一步。机器人还必须理解物体在哪里、其抓取器如何移动、接触发生时可能发生什么，以及哪种动作最有可能成功完成任务。

A world model helps the robot reason about these relationships. It can predict the visual result of an action, infer the action that caused a change, or generate an action to produce a desired outcome. Cosmos 3 Edge brings these capabilities together in one model, on-device. Its shared representation enables physical AI systems to understand the current world state, simulate possible futures, and connect those futures to actions. 世界模型帮助机器人推理这些关系。它可以预测动作的视觉结果、推断导致变化的动作，或生成动作以产生预期的结果。Cosmos 3 Edge 将这些能力整合到一个设备端模型中。其共享表示使物理 AI 系统能够理解当前的世界状态、模拟可能的未来，并将这些未来与动作联系起来。

Two Transformer Towers, One Shared Representation

双 Transformer 塔，一个共享表示

Cosmos 3 combines two transformer towers:

Autoregressive tower: Processes vision and text tokens for understanding and reasoning.
Diffusion tower: Processes vision, audio, and action tokens for prediction, generation, and neural simulation. Cosmos 3 结合了两个 Transformer 塔：
自回归塔 (Autoregressive tower)： 处理视觉和文本 Token，用于理解和推理。
扩散塔 (Diffusion tower)： 处理视觉、音频和动作 Token，用于预测、生成和神经模拟。

The two towers maintain separate normalization layers and multilayer perceptrons. They share multimodal attention layers, which align information across language, video, audio, and action. This design allows the model to reason about a scene before generating an output. Depending on the task, Cosmos 3 can produce reasoning tokens from the autoregressive tower or denoised video and action tokens from the diffusion tower. 这两个塔维护独立的归一化层和多层感知机。它们共享多模态注意力层，用于对齐语言、视频、音频和动作之间的信息。这种设计使模型能够在生成输出之前对场景进行推理。根据任务的不同，Cosmos 3 可以从自回归塔生成推理 Token，或从扩散塔生成去噪后的视频和动作 Token。

The attention pattern is also adapted to each type of information. Language uses causal attention, where each token attends to the tokens that came before it. Diffusion tokens attend more broadly to the available context, supporting coherent prediction and generation. Together, these components give the model a common representation of what is happening, what may happen next, and how an action could influence the outcome. 注意力模式也针对每种信息类型进行了调整。语言使用因果注意力，即每个 Token 只关注其之前的 Token。扩散 Token 则更广泛地关注可用上下文，从而支持连贯的预测和生成。这些组件共同为模型提供了一个关于“正在发生什么”、“接下来可能发生什么”以及“动作如何影响结果”的通用表示。

A Common Representation for Action

通用的动作表示

Physical systems describe actions differently. A vehicle may represent an action through ego pose and movement. A camera uses camera motion. A robot arm uses the pose of its end effector, while a hand or gripper also needs to represent grasp state. Cosmos 3 maps these different embodiments into a common action representation. 物理系统描述动作的方式各不相同。车辆可能通过自身位姿和移动来表示动作；摄像头使用摄像机运动；机械臂使用末端执行器的位姿，而手或抓取器还需要表示抓取状态。Cosmos 3 将这些不同的载体映射为一种通用的动作表示。

Actions are encoded as compact geometric vectors that capture:

Translation
Rotation
Manipulation state 动作被编码为紧凑的几何向量，捕捉以下信息：
平移
旋转
操作状态

This creates a direct connection between control and the visual structure of the world. The model can associate changes in pixels with physical motion, spatial relationships, and control inputs. As a result, generated video becomes more than a visual prediction. It can represent how the world is expected to change in response to an action. This gives developers training data grounded in motion, control, and cause and effect. 这在控制与世界的视觉结构之间建立了直接联系。模型可以将像素的变化与物理运动、空间关系和控制输入关联起来。因此，生成的视频不仅仅是视觉预测，它还能表示世界预计如何响应某个动作而发生变化。这为开发者提供了基于运动、控制以及因果关系的训练数据。

Policy Mode

策略模式

As a policy, Cosmos 3 predicts an action together with its expected visual consequence. The model can generate what the system should do and simulate what is likely to happen next. This connects world modeling directly to robot policy training and evaluation. Current state in, action and visual consequence out. These modes enable one model to learn cause, effect, and policy. Action can flow through the model in both directions, allowing Cosmos 3 Edge to predict the effects of an action or infer an action from its effects. 作为一种策略，Cosmos 3 可以预测动作及其预期的视觉后果。模型既能生成系统应该做什么，也能模拟接下来可能发生什么。这直接将世界建模与机器人策略训练和评估联系起来。输入当前状态，输出动作和视觉后果。这些模式使单一模型能够学习因果关系和策略。动作可以在模型中双向流动，使 Cosmos 3 Edge 既能预测动作的效果，也能从效果推断出动作。

We are also releasing Cosmos 3 Edge Policy (DROID): A robot manipulation policy post-trained on the DROID dataset for pick-and-place tasks, with accompanying post-training scripts. Developers can use a small cluster of H100 or NVIDIA DGX Station to efficiently fine-tune Cosmos 3 Edge for their target workloads before deploying to NVIDIA edge and accelerated computing platforms. 我们还发布了 Cosmos 3 Edge Policy (DROID)：这是一种在 DROID 数据集上针对抓取任务进行后训练的机器人操作策略，并附带了后训练脚本。开发者可以使用少量 H100 集群或 NVIDIA DGX Station，在部署到 NVIDIA 边缘和加速计算平台之前，高效地针对目标工作负载微调 Cosmos 3 Edge。

Post-trained samples for improved performance

用于提升性能的后训练样本

Alongside the base models, we’re releasing reference post-trained checkpoints and training recipes to help developers adapt and optimize Cosmos 3 for their own applications. Cosmos 3 is an open world foundation model platform. 除了基础模型外，我们还发布了参考后训练检查点和训练配方，以帮助开发者针对自己的应用调整和优化 Cosmos 3。Cosmos 3 是一个开放的世界基础模型平台。