Reinforcement Learning with Decomposed Subtasks
Reinforcement Learning with Decomposed Subtasks
基于子任务分解的强化学习
Abstract: Group Relative Policy Optimization (GRPO) and related policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before it enters the policy update. When the task composes distinct skills, especially under sparse and delayed environmental feedback, this collapsing is lossy: the optimizer must implicitly infer which competency drove the outcome and how that should change behavior.
摘要： 群体相对策略优化（GRPO）及相关用于训练语言模型智能体的策略梯度方法，在进入策略更新之前，会将整个多轮交互过程压缩为一个单一的标量轨迹奖励。当任务由多种不同技能组成，且环境反馈稀疏且滞后时，这种压缩是有损的：优化器必须隐式地推断出是哪种能力驱动了结果，以及应如何改变行为。
We argue the right primitive is not a better scalar but a decomposition: trajectory reward should be split along subtasks before it enters the policy update. We introduce Reinforcement Learning with Decomposed Subtasks (RLDS), whose core is Subtask-Decomposed Advantage Estimation (SDAE): a replacement for the scalar GRPO advantage that splits trajectory reward into per-subtask shares on a fixed taxonomy, computes a group-relative advantage per subtask, and distributes per-token credit by weighting each subtask’s advantage by its importance, concentrating it around the step where a reflection marks that subtask’s execution as consequential.
我们认为，正确的原语并非更好的标量，而是分解：轨迹奖励应在进入策略更新前按子任务进行拆分。我们引入了“基于子任务分解的强化学习”（RLDS），其核心是“子任务分解优势估计”（SDAE）：这是一种替代标量 GRPO 优势的方法，它根据固定的分类法将轨迹奖励拆分为各子任务的份额，计算每个子任务的群体相对优势，并通过各子任务的重要性对优势进行加权，从而分配每个 Token 的贡献度，并将其集中在标记该子任务执行具有决定性意义的步骤上。
We evaluate on four agentic benchmarks: FrozenLake (sparse grid navigation), HotpotQA (multi-hop QA, one retrieval tool), ScienceWorld (long-horizon embodied science), and DeepResearch (long-form research, four tools, composite rubric rubric reward). Heterogeneity diagnostics emitted during training show where decomposition pays off - gains scale with subtask heterogeneity, largest on the high-heterogeneity tasks ScienceWorld (+11.5 points, paired-bootstrap 95% CI [+9.8, +13.3]) and FrozenLake (+9.8 points, [+7.0, +12.8]), and within noise on HotpotQA and DeepResearch, where the diagnostics predicted little to recover.
我们在四个智能体基准测试上进行了评估：FrozenLake（稀疏网格导航）、HotpotQA（多跳问答，含一个检索工具）、ScienceWorld（长程具身科学）和 DeepResearch（长篇研究，含四个工具及复合评分准则）。训练期间输出的异质性诊断显示了分解的价值所在——收益随子任务异质性的增加而扩大，在异质性较高的任务 ScienceWorld（+11.5 分，配对自助法 95% 置信区间 [+9.8, +13.3]）和 FrozenLake（+9.8 分，[+7.0, +12.8]）上增幅最大；而在诊断预测收益较小的 HotpotQA 和 DeepResearch 上，结果则处于噪声范围内。
ScienceWorld is also more compute-efficient under RLDS than scalar GRPO (-10.9% wall-clock per step), as long rollouts amortize the fixed reflect-and-grade overhead.
在 RLDS 下，ScienceWorld 的计算效率也高于标量 GRPO（每步挂钟时间减少 10.9%），因为长序列交互摊销了固定的“反思与评分”开销。