Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation

诱导无需奖励的评估准则以减少智能体评估中的过度信任问题

Abstract: Evaluating language-model agents at scale increasingly relies on a second language model as an automatic judge, because the gold signal, an executable environment reward, is expensive, slow, or unavailable at deployment time. Such a judge is a reward-free proxy whose value depends on whether it can be trusted, yet existing judges either hand-write the scoring rubric, as in G-Eval, or fine-tune the judge’s weights, and both tend to credit fluent but unsuccessful trajectories as successes.

摘要： 在大规模评估语言模型智能体时，人们越来越依赖第二个语言模型作为自动评估器，因为作为“黄金标准”的可执行环境奖励在部署时往往成本高昂、速度缓慢或不可用。这种评估器是一种无需奖励的代理，其价值取决于是否值得信赖。然而，现有的评估方法要么像 G-Eval 那样手动编写评分准则，要么微调评估器的权重，但这两种方法往往会将流畅但失败的轨迹误判为成功。

We instead induce the text of an agent-judging rubric from a small set of ground-truth-labeled trajectories, grounding it in true outcomes. We present RubricForge, which evolves a judge rubric by reflective evolution against labeled trajectories to maximize agreement with the environment reward, freezes it, and applies it to held-out trajectories in one model call with no environment access. The optimized artifact is human-readable text, so every verdict is attributable to named criteria.

我们转而从一小部分带有真实标签的轨迹中诱导生成智能体评估准则文本，使其基于真实结果。我们提出了 RubricForge，它通过针对标记轨迹的反射式演化来优化评估准则，以最大化与环境奖励的一致性，随后将其冻结，并在无需访问环境的情况下，通过单次模型调用将其应用于留存轨迹。优化后的产物是人类可读的文本，因此每一个判定结果都可以归因于具体的命名标准。

Using one frozen 7B model as both agent and judge, on tau-bench (173 labeled trajectories drawn from 220 rollouts) and WebShop (160), the principal gain is faithfulness rather than raw agreement. The edge over a generic G-Eval judge is not statistically significant (McNemar p = 0.248), and absolute-score calibration marginally favors the generic judge (|err| difference -0.048, p = 2x10^-4). Yet RubricForge over-credits failed trajectories roughly half as often (0.115 vs. 0.173 false-pass rate on tau-bench, with three over-credit catches and zero reversals) and ranks graded WebShop outcomes more faithfully (Spearman 0.410 vs. 0.370).

在使用同一个冻结的 7B 模型同时作为智能体和评估器，并在 tau-bench（从 220 次运行中抽取的 173 条标记轨迹）和 WebShop（160 条）上进行测试时，主要收益在于忠实度而非原始一致性。与通用 G-Eval 评估器相比，其优势在统计学上并不显著（McNemar p = 0.248），且绝对分数校准略微偏向通用评估器（|err| 差异 -0.048，p = 2x10^-4）。然而，RubricForge 对失败轨迹的过度信任率大约降低了一半（在 tau-bench 上为 0.115 对比 0.173，且成功捕捉到三次过度信任情况，零次反转），并能更忠实地对 WebShop 的结果进行排序（Spearman 系数为 0.410 对比 0.370）。

For a reward-free evaluator the false-pass rate, not aggregate agreement, is the deployment-relevant quantity, since a false pass ships a broken agent whereas a false fail merely costs a retry.

对于无需奖励的评估器而言，部署时最相关的指标是“误通过率”而非“总体一致性”，因为误通过会导致部署一个有缺陷的智能体，而误失败仅仅意味着需要重试。