When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning

当不可信的 Token 被强化时：大语言模型强化学习中的尾部感知信用校准

Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. 强化学习（RL）在增强大语言模型（LLM）的推理能力方面取得了显著成功。然而，广泛使用的无评论员（critic-free）强化学习方法依赖于统一的信用分配，即无论 Token 之间存在何种差异，都将相同的优势（advantage）广播给所有 Token。

We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: low-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior. 我们识别出这种设计的一个关键失效模式，我们将其称为“正向信用污染”（Positive-Credit Contamination）：在同一轨迹中，语境上错误的低概率尾部 Token 会获得与合理 Token 相同的正向信用，导致对错误推理行为的无差别强化。

To mitigate this issue, we propose Tail-Aware Credit calibratiOn (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates. TACO first computes a tail-risk score that incorporates the local generation context to assess each token’s risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty-driven exploration. 为了缓解这一问题，我们提出了“尾部感知信用校准”（Tail-Aware Credit calibratiOn，简称 TACO），这是一种通过校准统一信用分配来抑制不良正向更新的方法。TACO 首先计算一个结合了局部生成语境的“尾部风险评分”，以评估每个 Token 落入不可靠尾部的风险，从而区分意外的稀有性与由不确定性驱动的探索。

TACO then uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened. 随后，TACO 利用该评分来调整风险 Token 的正向信用，且不会完全移除它们的梯度，从而使反复出现的有用稀有模式能够积累强化，同时逐渐抑制偶然的噪声。

Experimental results across three LLMs and eight benchmarks show that TACO consistently outperforms GRPO-style baselines. Notably, TACO improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: this https URL. 在三个大语言模型和八个基准测试上的实验结果表明，TACO 的表现始终优于 GRPO 风格的基线模型。值得注意的是，TACO 提高了训练稳定性，支持了长程强化学习中持续的性能提升。源代码地址：点击此处。