Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains

查看原文 / View Original 编辑 / Edit

Computer Science > Machine Learning arXiv:2610.26932 (cs) [Submitted on 17 Oct 2026] Title: Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains Authors: Ashish Bansal, Hossam Amer, Keerat Kaur Guliani, Parinita Edke, Lesley Matheson, Leizer Teran.

计算机科学 > 机器学习 arXiv:2610.26932 (cs) [提交于 2026 年 10 月 17 日] 标题：专家引导的评分标准作为非可验证领域强化学习的密集奖励信号作者：Ashish Bansal, Hossam Amer, Keerat Kaur Guliani, Parinita Edke, Lesley Matheson, Leizer Teran。

Abstract: Large language models (LLMs) are increasingly deployed as autonomous agents in non-verifiable domains such as financial services, where they must interpret user intent, rewrite queries, and retrieve information under domain-specific constraints. Although reinforcement learning (RL) can train such agents from the consequences of their actions, recent successes largely rely on verifiable tasks with automatically computed binary rewards.

摘要：大型语言模型 (LLM) 正越来越多地作为自主智能体部署在金融服务等非可验证领域，在这些领域中，它们必须在特定领域的约束下解释用户意图、重写查询并检索信息。尽管强化学习 (RL) 可以通过智能体行为的后果来训练它们，但近期的成功主要依赖于具有自动计算二元奖励的可验证任务。

In non-verifiable retrieval tasks, assessing query quality requires scarce subject-matter expert (SME) judgment that is inherently graded rather than binary. With Group Relative Policy Optimization (GRPO), sparse rewards yield no useful learning signal when all sampled outputs within a group receive the same outcome, causing optimization to stall.

在非可验证的检索任务中，评估查询质量需要稀缺的主题专家 (SME) 判断，这种判断本质上是分级的，而非二元的。在使用组相对策略优化 (GRPO) 时，如果组内所有采样输出的结果相同，稀疏奖励将无法产生有用的学习信号，从而导致优化停滞。

We introduce a method that converts a small, fixed investment of SME judgment into a dense reward and combines it with retrieval-based metrics to form a hybrid, GRPO-compatible training signal. Experiments show improvements of up to 6% in Recall@3 and 6.7% in MRR@3 compared with sparse-reward baselines.

我们引入了一种方法，将少量、固定的 SME 判断投入转化为密集奖励，并将其与基于检索的指标相结合，形成一种混合的、与 GRPO 兼容的训练信号。实验表明，与稀疏奖励基线相比，Recall@3 提高了 6%，MRR@3 提高了 6.7%。