ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

ASK in the Dark：部分可观测环境下的不确定性门控大模型辅助

Abstract: Reinforcement learning agents operating under partial observability must act on incomplete information, making them natural candidates for guidance from small language models (SLMs) that carry broad reasoning priors. Yet integrating SLM guidance into this setting has proven difficult: across all test environments, vanilla uncertainty-gated approaches achieve an overwrite rate at or near zero, meaning the SLM almost never contributes an independent action.

摘要： 在部分可观测环境下运行的强化学习智能体必须基于不完整的信息采取行动，这使其成为利用具备广泛推理先验的小型语言模型（SLM）进行引导的天然候选者。然而，将 SLM 引导集成到该设置中已被证明十分困难：在所有测试环境中，传统的“不确定性门控”方法实现的覆盖率（overwrite rate）几乎为零，这意味着 SLM 几乎从未贡献过独立的行动。

We trace this failure to the bare egocentric prompt, which provides insufficient context for genuine reasoning, and identify it as a context problem rather than a capacity problem. We propose ASK+, which supplies the SLM with trajectory-aware context (a partially revealed map, visited positions, and action history) and structured chain-of-thought reasoning, converting it from a passive redundancy check into a more informative consultant that occasionally corrects the policy.

我们将这一失败归因于简单的自我中心提示词（egocentric prompt），它无法为真正的推理提供足够的上下文，并将其识别为上下文问题而非模型能力问题。我们提出了 ASK+，它为 SLM 提供了轨迹感知上下文（部分揭示的地图、已访问位置和行动历史）以及结构化的思维链推理，将其从被动的冗余检查转变为一个更具信息量的顾问，能够偶尔纠正策略。

We further establish that the predictive entropy signal used for selective querying measures action uncertainty rather than state uncertainty and remains informative in POMDPs, making uncertainty-gated assistance viable beyond fully observable settings. The stateful prompt drives substantial gains: on DoorKey, where vanilla ASK matches PPO (both 89%), ASK+ reaches 93% success; on FourRooms, success climbs from 53% to 70%; on HigherLower, accuracy reaches 73.7%, matching the SLM-only upper bound.

我们进一步证实，用于选择性查询的预测熵信号衡量的是行动不确定性而非状态不确定性，并且在部分可观测马尔可夫决策过程（POMDPs）中仍然具有信息价值，这使得不确定性门控辅助在完全可观测环境之外依然可行。这种有状态的提示词带来了显著的性能提升：在 DoorKey 环境中，传统 ASK 与 PPO 持平（均为 89%），而 ASK+ 的成功率达到了 93%；在 FourRooms 中，成功率从 53% 提升至 70%；在 HigherLower 中，准确率达到 73.7%，达到了仅使用 SLM 的性能上限。

Across all environments, Qwen3.5-2B matches or exceeds Qwen3.5-4B, confirming that prompt design and selective gating dominate the impact of model scale, enabling guidance without large models.

在所有环境中，Qwen3.5-2B 的表现均持平或超过了 Qwen3.5-4B，这证实了提示词设计和选择性门控对性能的影响超过了模型规模本身，从而实现了无需大型模型即可进行有效引导的目标。