Process Reward Informed Tree Rollout for Effective Multi-Turn RL
Process Reward Informed Tree Rollout for Effective Multi-Turn RL
基于过程奖励引导的树状采样：实现高效的多轮强化学习
Reinforcement learning (RL) has become a key approach for training LLM agents, yet popular methods such as GRPO/RLOO rely on multiple independently sampled complete trajectories for advantage estimation. 强化学习（RL）已成为训练大语言模型（LLM）智能体的关键方法，然而，诸如 GRPO/RLOO 等主流方法依赖于多次独立采样的完整轨迹来进行优势估计。
In long-horizon agentic tasks, such a uniform rollout strategy can waste budget on uninformative dead-end attempts, while promising intermediate states do not receive sufficient exploration. 在长程智能体任务中，这种统一的采样策略往往会将计算预算浪费在无意义的死胡同尝试上，而有潜力的中间状态却无法获得充分的探索。
The multi-turn structure of agentic trajectories, with interleaved actions and observations, naturally supports organizing a trajectory group as a tree, where each turn serves as a decision point for exploration. 智能体轨迹的多轮结构（包含交替的动作与观测）天然支持将轨迹组组织为树状结构，其中每一轮都可以作为探索的决策点。
This perspective reframes effective exploration as the problem of deciding where to branch. We propose Process-Scorer Guided Adaptive Tree Rollout (PATR), a quality-aware rollout framework for multi-turn agent RL. 这一视角将高效探索重新定义为“决定在何处进行分支”的问题。我们提出了“过程评分引导的自适应树状采样”（PATR），这是一个面向多轮智能体强化学习的质量感知采样框架。
PATR uses task-appropriate process feedback to score partial trajectories, selectively branches from promising states, reuses shared prefixes, and conservatively stops degenerate paths to reduce wasted sampling. PATR 利用任务相关的过程反馈对部分轨迹进行评分，从有潜力的状态选择性地进行分支，复用共享的前缀，并保守地停止退化路径，从而减少采样浪费。
The resulting rollout groups remain compatible with standard policy optimization while providing more efficient exploration under the same training budget. 由此产生的采样组不仅与标准策略优化兼容，还能在相同的训练预算下提供更高效的探索。
We evaluate PATR on FrozenLake and the challenging SWE-Bench, which is largely unexplored by prior tree-rollout agent RL methods. 我们在 FrozenLake 和极具挑战性的 SWE-Bench 上评估了 PATR，后者在以往的树状采样智能体强化学习研究中鲜有涉及。
Experiments show that PATR improves performance by up to +5.0 points on SWE-Bench and +9.3 points on FrozenLake, highlighting process-guided tree rollouts as an effective strategy for scalable multi-turn RL. 实验表明，PATR 在 SWE-Bench 上提升了最高 +5.0 分，在 FrozenLake 上提升了 +9.3 分，这凸显了过程引导的树状采样是实现可扩展多轮强化学习的一种有效策略。