A Beginner’s Guide to World Models

世界模型入门指南

Intro In Machine Learning, a World Model is a system that builds an internal representation of an environment and predicts how that environment changes over time in response to actions. Simply put, it’s an AI model that simulates the world’s dynamics (i.e. physics, object interactions, causality). These days, World Models are being used to power robots, autonomous driving, and interactive video generation because they allow an AI to “think before it acts” by running mental simulations of reality.

简介在机器学习领域，世界模型（World Model）是一种能够构建环境内部表征，并预测环境如何随动作而随时间变化的系统。简单来说，它是一种模拟世界动态（即物理规律、物体交互、因果关系）的 AI 模型。如今，世界模型正被广泛应用于机器人、自动驾驶和交互式视频生成，因为它们允许 AI 通过运行现实的心理模拟来“三思而后行”。

Early ideas date to the 1990s, when German researchers designed Recurrent Neural Networks that predict future states from observations and used those predictions to train agents without constant real-world trial and error. Then, in 2022, Yann LeCun revived the idea of autonomous machine intelligence, saying that intelligence requires predictive models of the world rather than pure pattern matching. The first World Model proposed by LeCun was the Joint Embedding Predictive Architecture (JEPA) with the idea that giving an internal representation of how the world functions to a self-supervised model, it could improve its results. The key idea is that the AI learns how the physical environment (the “world”) works by predicting abstract concepts in a continuous space, rather than guessing the next token (e.g. pixel or word). For this reason, JEPA is very different from Transformer architectures, and it’s considered the foundation for World Models.

早期的构想可以追溯到 20 世纪 90 年代，当时德国研究人员设计了循环神经网络（RNN），通过观察预测未来状态，并利用这些预测来训练智能体，从而避免了持续的现实世界试错。随后，在 2022 年，Yann LeCun 重提了自主机器智能的概念，指出智能需要对世界进行预测建模，而非单纯的模式匹配。LeCun 提出的第一个世界模型是联合嵌入预测架构（JEPA），其核心思想是为自监督模型提供关于世界运作方式的内部表征，从而提升其结果。关键在于，AI 通过在连续空间中预测抽象概念来学习物理环境（即“世界”）的运作方式，而不是猜测下一个标记（如像素或单词）。因此，JEPA 与 Transformer 架构有很大不同，被视为世界模型的基石。

In 2024, Google introduced its World Model Genie, a research prototype that lets you create and explore infinitely diverse worlds. California’s autonomous taxi Waymo adopted Google’s Genie to create its own specialized model for self-driving simulation (Waymo World Model).

2024 年，谷歌推出了世界模型 Genie，这是一个研究原型，允许用户创建并探索无限多样的世界。加州的自动驾驶出租车公司 Waymo 采用了谷歌的 Genie，为其自动驾驶模拟创建了专门的模型（Waymo 世界模型）。

State of the Art In 2026, the startup World Labs released Marble, an AI with “spatial intelligence” that can create interactive and explorable 3D environments from text prompts. Similarly, Alibaba launched its World Model Happy Oyster, designed for world-building based on text and image prompts. Then, the latest World Model was released in June 2026 by Nvidia: Cosmos, a family of open-weight models that combine physical reasoning, world simulation, and action generation. Those three World Models represent the current state-of-the-art.

技术前沿 2026 年，初创公司 World Labs 发布了 Marble，这是一款具备“空间智能”的 AI，能够根据文本提示创建可交互、可探索的 3D 环境。同样，阿里巴巴推出了其世界模型 Happy Oyster，旨在基于文本和图像提示进行世界构建。随后，英伟达于 2026 年 6 月发布了最新的世界模型：Cosmos，这是一个结合了物理推理、世界模拟和动作生成的开源权重模型系列。这三个世界模型代表了当前的技术前沿。

For example, I’ll write a prompt on Alibaba’s Happy Oyster to generate an environment. And just from this simple prompt, the model creates a confined 3D world that you can explore for a limited time (2 min) in a sort of video game style.

例如，我会在阿里巴巴的 Happy Oyster 上输入一个提示词来生成一个环境。仅凭这个简单的提示，模型就能创建一个受限的 3D 世界，你可以像玩电子游戏一样在其中探索有限的时间（2 分钟）。

Architectures Basically, World Models are systems that build an internal simulation of an environment to predict its future states. They can be categorized into four architectures:

架构从本质上讲，世界模型是构建环境内部模拟以预测其未来状态的系统。它们可以分为四种架构：

Joint Embedding Predictive Architectures (JEPA) eliminate the need for generative decoders and prevent the model from wasting computational power on unpredictable background details (like flickering leaves) by forcing it to focus only on large-scale physical dynamics. Use-case: Meta’s V-JEPA.
联合嵌入预测架构 (JEPA)：消除了对生成式解码器的需求，并通过强制模型仅关注大规模物理动态，防止其在不可预测的背景细节（如摇曳的树叶）上浪费计算能力。应用案例：Meta 的 V-JEPA。
Recurrent Stochastic State Models (RSSM) process sequential data by breaking it into a belief state (long-term memory) and a stochastic state (to capture environmental randomness and uncertainty). Use-case: open-source project Dreamer.
循环随机状态模型 (RSSM)：通过将序列数据分解为信念状态（长期记忆）和随机状态（捕捉环境的随机性和不确定性）来处理数据。应用案例：开源项目 Dreamer。
Tree-Search Models, instead of predicting what the next frame looks like, predict the next hidden state, the action value, and the immediate reward, integrating a Monte Carlo Tree Search to plan ahead. Use-case: Google’s MuZero.
树搜索模型：不预测下一帧的样子，而是预测下一个隐藏状态、动作价值和即时奖励，并集成蒙特卡洛树搜索来进行前瞻性规划。应用案例：谷歌的 MuZero。
Generative Foundation Simulators use autoregressive Diffusion Transformers trained on massive internet-scale video datasets, and act as real-time generative video game engines. They take text, image, or action prompts and generate temporally consistent, physically plausible photorealistic future frames. Use-case: Google’s Genie and Nvidia’s Cosmos.
生成式基础模拟器：使用在海量互联网规模视频数据集上训练的自回归扩散 Transformer，充当实时生成式视频游戏引擎。它们接收文本、图像或动作提示，并生成时间一致、物理上合理且逼真的未来帧。应用案例：谷歌的 Genie 和英伟达的 Cosmos。

Build a World Model The easiest and most accessible type of World Model is SGF (Simple, Good, Fast). It’s a simplified non-recurrent latent dynamics model where spatio-temporal data is compressed into a hidden layer (latent space) without using traditional recurrent loops (like LSTMs). I’m going to build a World Model inspired by the SGF approach, combined with the general latent-dynamics architecture introduced in this paper, which adds a decoder layer to reconstruct the prediction as an image.

构建世界模型 最简单且易于上手的一类世界模型是 SGF（Simple, Good, Fast）。它是一种简化的非循环潜在动力学模型，将时空数据压缩到隐藏层（潜在空间）中，而不使用传统的循环回路（如 LSTM）。我将构建一个受 SGF 方法启发的世界模型，并结合本文介绍的通用潜在动力学架构，该架构增加了一个解码器层，将预测结果重构为图像。

For the environment, let’s use the Atari game BattleZone (pip install ale-py) and the famous Gymnasium library (pip install gymnasium).

对于环境，我们使用 Atari 游戏 BattleZone (pip install ale-py) 和著名的 Gymnasium 库 (pip install gymnasium)。

(Code snippets omitted for brevity)

The environment is a discrete space of 18 possible actions (go right, go left, attack…), therefore the model will have to learn meaningful spatial dynamics: your tank moves, enemies appear in different directions, projectiles travel, and the camera changes.

该环境是一个包含 18 种可能动作（向右、向左、攻击等）的离散空间，因此模型必须学习有意义的空间动态：你的坦克移动、敌人在不同方向出现、炮弹飞行以及摄像机视角的切换。

(Code snippets omitted for brevity)

I will build a World Model Neural Network using PyTorch (pip install torch). The input data is Atari RGB pictures (210x160x3) with raw pixel values (0-255). The first step is cleaning and reshaping the data into a small grayscale tensor (1x64x64) with normalized values (0-1), so the Neural Network can digest it efficiently.

我将使用 PyTorch (pip install torch) 构建一个世界模型神经网络。输入数据是 Atari RGB 图片（210x160x3），原始像素值为 0-255。第一步是清理数据并将其重塑为小的灰度张量（1x64x64），并将值归一化（0-1），以便神经网络能够高效处理。

(Code snippets omitted for brevity)

Now that we can process data, we should start our data collection by building a memory of the game’s experiences. The collection loop runs the game using random actions and stores each example (obs, action, next_obs) in the Replay buffer. So after this loop, we have a dataset of how the environment behaves: “I saw this image…”

现在我们已经可以处理数据了，接下来应该通过构建游戏体验的记忆来开始数据收集。收集循环使用随机动作运行游戏，并将每个示例（观察值、动作、下一个观察值）存储在重放缓冲区中。因此，在循环结束后，我们就拥有了一个关于环境行为的数据集：“我看到了这张图像……”