In-Context Reinforcement Learning under Non-Stationarity: A Survey

非平稳环境下的上下文强化学习：综述

The development of decision-pretrained transformers, algorithm distillation, long-context meta-RL, and retrieval-augmented agents has renewed interest in in-context reinforcement learning (ICRL): the ability of a pretrained or fine-tuned decision model to infer latent task rules and improve future behavior from interaction context, without test-time parameter updates.

决策预训练 Transformer、算法蒸馏、长上下文元强化学习（meta-RL）以及检索增强智能体的发展，重新激发了人们对上下文强化学习（ICRL）的兴趣：即预训练或微调后的决策模型在无需测试时参数更新的情况下，通过交互上下文推断潜在任务规则并改善未来行为的能力。

This line of work asks when trial-and-error evidence, rewards, transitions, demonstrations, feedback, or retrieved experience can make learning-like computation happen inside the context window. However, existing surveys of ICRL mainly organize the field around pretraining objectives, architectures, context formats, evaluation protocols, and theoretical mechanisms, while the non-stationary setting remains comparatively underexamined.

这一研究方向探讨了试错证据、奖励、状态转移、演示、反馈或检索到的经验在何时能够使上下文窗口内发生类似学习的计算。然而，现有的 ICRL 综述主要围绕预训练目标、架构、上下文格式、评估协议和理论机制来组织该领域，而对非平稳环境设置的研究相对不足。

In changing environments, accumulated context is not merely more evidence about a fixed task: the reward specification, transition kernel, observation channel, action interface, constraint model, or demonstration and memory distribution can fall out of alignment with the current regime. Previously useful context can therefore become stale, misleading, or useful again when an old regime returns.

在不断变化的环境中，积累的上下文不仅仅是关于固定任务的更多证据：奖励规范、状态转移核、观测通道、动作接口、约束模型或演示与记忆分布可能会与当前机制不再一致。因此，先前有用的上下文可能会变得陈旧、具有误导性，或者在旧机制回归时重新变得有用。

We survey non-stationary ICRL as the problem of adapting through context while deployed policy parameters remain fixed: the policy must infer both the current decision rule and which parts of its accumulated evidence still support that rule.

我们将非平稳 ICRL 视为在部署策略参数保持不变的情况下，通过上下文进行适应的问题：策略必须同时推断出当前的决策规则，以及其积累的证据中哪些部分仍然支持该规则。

We define non-stationary ICRL, relate it to meta-RL, decision sequence modeling, retrieval-augmented RL, value- and model-aware ICRL, and reward-feedback agents, and organize the literature along three questions: what changes, how the change unfolds, and how observable the change is to the agent.

我们定义了非平稳 ICRL，将其与元强化学习、决策序列建模、检索增强强化学习、价值与模型感知 ICRL 以及奖励反馈智能体联系起来，并围绕三个问题对文献进行了梳理：发生了什么变化、变化如何展开，以及智能体对变化的感知程度如何。