Learning Stateful Predictive Knowledge From Experience

从经验中学习状态化预测知识

As large language model (LLM) agents increasingly learn from experience, they primarily rely on trajectory-level reflection to extract insights. Viewed through the lens of predictive knowledge, we argue that this approach operates on episodic hindsight rather than predictive foresight, yielding brittle, path-dependent heuristics.

随着大语言模型（LLM）智能体越来越多地从经验中学习，它们主要依赖轨迹层面的反思来提取洞察。从预测知识的角度来看，我们认为这种方法基于情境化的事后回顾，而非预测性的前瞻，从而导致了脆弱且依赖路径的启发式策略。

To address this, we propose Stateful Knowledge Learning (SKL). SKL shifts the agent’s focus from trajectory-level summarization to maintaining Stateful Knowledge: explicit, declarative predictive assessments anchored to state.

为了解决这一问题，我们提出了状态化知识学习（Stateful Knowledge Learning, SKL）。SKL 将智能体的关注点从轨迹层面的总结转移到维护“状态化知识”上：即锚定于特定状态的、显式的、声明式的预测评估。

We first demonstrate a motivating example showing how stateful knowledge provides granularity, enhances generalization, and enables knowledge bootstrapping. To further scale up the idea, we introduce two algorithms via self-distillation (SKL-SD) and reinforcement learning (SKL-RL), training agents to autonomously extract state-grounded predictive knowledge from experience and learn to leverage it for policy making.

我们首先展示了一个激励性示例，说明了状态化知识如何提供粒度、增强泛化能力并实现知识引导（bootstrapping）。为了进一步扩展这一理念，我们引入了两种算法：自蒸馏（SKL-SD）和强化学习（SKL-RL），训练智能体从经验中自主提取基于状态的预测知识，并学习如何利用这些知识进行决策。

Experiments on interactive environments (WebShop, ScienceWorld) and a complex reasoning task (ChessPuzzles) demonstrate that equipping models with the inherent ability to learn stateful predictive knowledge significantly outpaces current reflection-based training paradigms.

在交互式环境（WebShop、ScienceWorld）和复杂推理任务（ChessPuzzles）上的实验表明，赋予模型学习状态化预测知识的内在能力，其表现显著优于当前基于反思的训练范式。