Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement
Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement
通用智能体迭代：一种用于迭代策略改进与递归自我改进的统一形式化框架
Abstract: When we speak of recursive self-improvement (RSI), are we speaking of a phenomenon, a mechanism, or a prospect? Towards autonomous and evolving intelligence, RSI is being claimed at many scales, while no single framework that formally describes these emerging instances exists. 摘要： 当我们谈论递归自我改进（RSI）时，我们指的是一种现象、一种机制，还是一种前景？在迈向自主与进化智能的过程中，RSI 在多个尺度上被提及，但目前尚无单一框架能够形式化地描述这些涌现出的实例。
Its counterpart in the classical realm, iterative policy improvement, is characterized by generalized policy iteration (GPI), a framework of broad applicability with well-understood theoretical properties, but only where the update principle and the evaluation base lie outside the agent. 其在经典领域对应的概念是迭代策略改进，通常由广义策略迭代（GPI）来表征。GPI 是一个具有广泛适用性且理论性质明确的框架，但其前提是更新原则和评估基准必须位于智能体之外。
In this paper, we propose Generalized Agent Iteration (GAI), a formal framework that describes iterative policy improvement and RSI as two cases of a single learning paradigm. GAI defines the agent as a configuration of modifiable components within a system and models the learning process as a cycle of agent evaluation and agent improvement. 在本文中，我们提出了通用智能体迭代（GAI），这是一个将迭代策略改进和 RSI 描述为单一学习范式下两种情况的形式化框架。GAI 将智能体定义为系统内可修改组件的配置，并将学习过程建模为智能体评估与智能体改进的循环。
Two pivotal dials then distinguish the instances: whether the improving mechanism is part of the agent and whether the standard it is measured against is grounded outside it. The former dial delineates the boundary between GPI and RSI, and the latter determines a system’s polarity as anchored, goal drift, or fully self-referential. 两个关键的调节维度区分了这些实例：改进机制是否属于智能体的一部分，以及其衡量标准是否建立在智能体之外。前者界定了 GPI 与 RSI 之间的边界，而后者则决定了系统的极性，即它是“锚定的”、“目标漂移的”还是“完全自指的”。
Moreover, we use these coordinates to place existing systems on the same two axes and make the defects of recursive self-improvement statable one condition at a time. We see this paper as a first step toward exploring a formal characterization of RSI that rests on the classical account, makes existing systems comparable, and provides a principled basis for analyzing and designing new ones. 此外，我们利用这些坐标将现有系统置于相同的两个轴上，从而能够逐一阐述递归自我改进的缺陷。我们将本文视为探索 RSI 形式化表征的第一步，该表征建立在经典理论之上，使现有系统具有可比性，并为分析和设计新系统提供了原则性基础。