The Artificial Experimentalist: Discovery and Control of Self-Organizing Phenomena with Autotelic Reinforcement Learning
The Artificial Experimentalist: Discovery and Control of Self-Organizing Phenomena with Autotelic Reinforcement Learning
人工实验者：利用自主目标强化学习发现并控制自组织现象
Abstract: Existing methods for exploring cellular automata and other complex systems mostly operate in open loop: they set initial conditions, execute a full simulation, and observe the outcome, without intervening during execution. 摘要： 现有的探索元胞自动机和其他复杂系统的方法大多以开环方式运行：即设定初始条件，执行完整模拟，然后观察结果，而不在执行过程中进行干预。
We introduce a closed-loop framework based on autotelic reinforcement learning, in which an agent autonomously samples diverse goals and learns a goal-conditioned policy to intervene in a complex system through minimal, local perturbations. 我们引入了一种基于“自主目标强化学习”（autotelic reinforcement learning）的闭环框架。在该框架中，智能体能够自主采样多样化的目标，并学习一种以目标为条件的策略，通过最小化的局部扰动来干预复杂系统。
We instantiate this framework on Lenia, a continuous cellular automaton known for life-like self-organizing patterns, in an agentic system we call CARL, and demonstrate three capabilities. 我们将此框架应用于 Lenia（一种以类生命自组织模式而闻名的连续元胞自动机），构建了一个名为 CARL 的智能体系统，并展示了其三项能力。
First, CARL discovers stable solitons across a wide range of Lenia update rules at a higher rate than heuristic baselines. 首先，CARL 在各种 Lenia 更新规则中发现稳定孤子（solitons）的效率高于启发式基准方法。
Second, it learns to steer the movement direction of existing solitons with few interventions, showing that CARL can control self-organizing patterns, not only create them. 其次，它学会了通过少量干预来引导现有孤子的运动方向，这表明 CARL 不仅能创造自组织模式，还能对其进行控制。
Third, humans can use trained agents to guide solitons through maze environments in real time by specifying high-level directional commands that the agent translates into low-level interventions. 第三，人类可以通过指定高级方向指令，利用训练好的智能体实时引导孤子穿过迷宫环境，而智能体会将这些指令转化为底层的干预动作。
Trained across diverse goals, update rules, and random initial states, the agents acquire policies that generalize zero-shot to various out-of-distribution conditions. 通过在多样化的目标、更新规则和随机初始状态下进行训练，智能体获得了能够对各种分布外（out-of-distribution）条件进行零样本泛化的策略。
These results suggest a path toward artificial experimentalist agents that, autonomously or with human guidance, discover and control emergent phenomena in complex systems. 这些结果为“人工实验者”智能体指明了方向，即它们能够自主或在人类指导下，发现并控制复杂系统中的涌现现象。