Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents

查看原文 / View Original 编辑 / Edit

Computer Science > Artificial Intelligence arXiv:2610.12598 (cs) [Submitted on 8 Oct 2026] Title: Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents Authors: Yiran Wu, Andrew Zhao, Yurui Chang, Qingyun Wu, Xuan Feng.

计算机科学 > 人工智能 arXiv:2610.12598 (cs) [提交于 2026 年 10 月 8 日] 标题：超越二元可验证奖励：软件工程智能体的行为自奖励机制作者：Yiran Wu, Andrew Zhao, Yurui Chang, Qingyun Wu, Xuan Feng。

Abstract: Reinforcement learning with verifiable rewards (RLVR) provides reliable supervision for software engineering agents through executable tests, but binary rewards fail to distinguish trajectories with substantially different debugging quality. We investigate whether the current policy can judge its own trajectories and use these judgments to improve task performance without a separately trained reward model.

摘要：带有可验证奖励的强化学习 (RLVR) 通过可执行测试为软件工程智能体提供了可靠的监督，但二元奖励无法区分调试质量存在显著差异的轨迹。我们研究了当前策略是否能够判断其自身的轨迹，并利用这些判断在无需额外训练奖励模型的情况下提升任务性能。

To this end, we introduce Behavioral Self-Rewarding (BSR), a reward-refinement method that reuses the policy to compare observable debugging behavior among trajectories from the same task with the same verifier outcome. BSR aggregates pairwise preferences into task-local scores and converts them into bounded auxiliary rewards, introducing reward contrast while preserving the verifier-defined pass/fail ordering.

为此，我们引入了行为自奖励 (BSR)，这是一种奖励细化方法，它复用策略来比较同一任务中具有相同验证器结果的轨迹之间的可观察调试行为。BSR 将成对偏好聚合为任务局部得分，并将其转换为有界的辅助奖励，在保留验证器定义的通过/失败排序的同时引入了奖励对比。

This provides additional supervision from already-collected trajectories without requiring new environment interactions. Experiments on SWE-bench Verified and SWE-bench-Live show consistent improvements across the evaluated Qwen3-14B and Qwen3-32B configurations, supporting using the policy’s own comparative judgments as a complementary learning signal for training software engineering agents.

这在无需新的环境交互的情况下，从已收集的轨迹中提供了额外的监督。在 SWE-bench Verified 和 SWE-bench-Live 上的实验表明，在评估的 Qwen3-14B 和 Qwen3-32B 配置中均有持续的改进，这支持将策略自身的比较判断作为训练软件工程智能体的补充学习信号。