Visualizing the World Model in an Agent's Value Function

Title: Visualizing the World Model in an Agent’s Value Function

标题：在智能体的价值函数中可视化世界模型

Computer Science > Artificial Intelligence arXiv:2610.14186 (cs) [Submitted on 9 Oct 2026] Title:Visualizing the World Model in an Agent’s Value Function Authors:Shawheen Naderi, Fatemeh Akherati, Yubei Chen View a PDF of the paper titled Visualizing the World Model in an Agent’s Value Function, by Shawheen Naderi and 2 other authors View PDF HTML (experimental)

计算机科学 > 人工智能 arXiv:2610.14186 (cs) [提交于 2026 年 10 月 9 日] 标题：在智能体的价值函数中可视化世界模型 作者：Shawheen Naderi, Fatemeh Akherati, Yubei Chen 查看由 Shawheen Naderi 及其他 2 位作者撰写的题为《在智能体的价值函数中可视化世界模型》的论文 PDF，查看 PDF HTML（实验性）

Abstract:Could model-free reinforcement learning algorithms actually be learning a model of the environment? In this work, we apply counterfactual interventions which hold everything about a state fixed, besides a targeted change, to explore if the value function of a Proximal Policy Optimization agent forms a world model of the Crafter environment.

摘要：无模型强化学习算法是否实际上在学习环境模型？在这项工作中，我们应用反事实干预，即在保持状态的所有其他方面不变的情况下，仅对目标变量进行更改，以探索近端策略优化（PPO）智能体的价值函数是否构成了 Crafter 环境的世界模型。

Despite only being trained to optimize reward, we discover that the value function learns concepts such as the rules governing when a diamond is collectible, water and food having an impact on health, heuristics regarding inventory, and more. Finally, we discuss the safety implications of our visualizations and some shortcomings of current reinforcement learning agents that will prevent us from building truly advanced machine intelligence.

尽管仅接受过优化奖励的训练，我们发现价值函数学习到了诸如钻石何时可收集的规则、水和食物对健康的影响、关于库存的启发式方法等概念。最后，我们讨论了这些可视化结果的安全意义，以及当前强化学习智能体在构建真正先进的机器智能方面所面临的一些阻碍。