PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation

Abstract: Retrieval-Augmented Generation enhances Large Language Models by grounding responses in external knowledge, but multi-hop reasoning remains vulnerable to error propagation, where early retrieval failures confound subsequent steps.

摘要： 检索增强生成（RAG）通过将回答建立在外部知识的基础上，增强了大语言模型的能力。然而，多跳推理仍然容易受到错误传播的影响，即早期的检索失败会干扰后续的推理步骤。

Standard outcome-based optimization only rewards the final answer, leaving intermediate retrieval and reasoning errors undetected. While existing process-based methods introduce step-level signals, they still score each step against the final answer, rewarding spurious successes where flawed retrieval coincidentally produces the correct answer.

标准的基于结果的优化仅对最终答案进行奖励，导致中间的检索和推理错误无法被检测到。虽然现有的基于过程的方法引入了步骤级信号，但它们仍然根据最终答案来评估每一步，这会奖励那些因检索错误却巧合得出正确答案的“虚假成功”。

Step-level supervision in RAG requires evaluating both logical validity and evidential grounding at each step. We introduce PRO-STEP: we train a generative PRM that evaluates both dimensions, employ PRM-guided value tree search to construct preference pairs contrasting valid steps against flawed ones, and optimize the policy via step-level Direct Preference Optimization.

RAG 中的步骤级监督需要同时评估每一步的逻辑有效性和证据基础。我们引入了 PRO-STEP：我们训练了一个能够评估这两个维度的生成式过程奖励模型（PRM），利用 PRM 引导的价值树搜索来构建对比有效步骤与缺陷步骤的偏好对，并通过步骤级的直接偏好优化（DPO）来优化策略。

Experiments on single and multi-hop QA datasets demonstrate that PRO-STEP achieves the best average EM and F1 across five benchmarks. Code, models, and training data are publicly available at this https URL.

在单跳和多跳问答数据集上的实验表明，PRO-STEP 在五个基准测试中均取得了最佳的平均 EM 和 F1 分数。代码、模型和训练数据已在链接中公开。