Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks

面向长程智能体任务的进度条件组策略优化

Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards by comparing trajectories or steps within a group. However, on difficult long-horizon tasks, this comparison can suffer from a sampling imbalance: repeated or low-effect actions dominate the high-probability region of the policy while useful state-changing actions remain under-sampled.

基于组的策略优化（Group-based policy optimization）正越来越多地被用于训练大语言模型（LLM）智能体，通过比较组内的轨迹或步骤，从稀疏的结果奖励中进行学习。然而，在困难的长程任务中，这种比较可能会遭遇采样不平衡问题：重复或低效的动作占据了策略的高概率区域，而有用的状态改变动作却未能得到充分采样。

This imbalance produces many all-failed rollout groups, where outcome rewards provide no direction for correcting the policy. Together, these effects can form a self-reinforcing credit trap: failure-dominated sampling yields no outcome-based correction, allowing repeated low-effect actions to persist.

这种不平衡会导致许多“全失败”的采样组出现，在这些组中，结果奖励无法为策略修正提供任何指导。这些因素共同构成了一种自我强化的“信用陷阱”（credit trap）：以失败为主导的采样无法产生基于结果的修正，从而导致重复的低效动作持续存在。

To break this loop, we propose Progress-conditioned Group Policy Optimization (ProGPO), which uses first-visit observation coverage only when all samples in a group receive zero outcome reward. Specifically, within such groups, ProGPO assigns higher relative advantages to trajectories or steps that visit more new states since reaching new observations is a prerequisite for task success.

为了打破这一循环，我们提出了“进度条件组策略优化”（ProGPO）。当组内所有样本获得的结果奖励均为零时，该方法会利用“首次访问观察覆盖率”（first-visit observation coverage）来引导优化。具体而言，在这些组内，ProGPO 会为访问了更多新状态的轨迹或步骤分配更高的相对优势，因为触达新的观察结果是任务成功的先决条件。

Experiments on two challenging agentic benchmarks, ALFWorld and WebShop with Qwen2.5-1.5/7B-Instruct, show that ProGPO consistently improves over group-based baselines, with particularly large gains on hard tasks.

在 ALFWorld 和 WebShop 这两个具有挑战性的智能体基准测试中，使用 Qwen2.5-1.5/7B-Instruct 模型进行的实验表明，ProGPO 在性能上始终优于基于组的基准方法，并且在困难任务上的提升尤为显著。