trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories
trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories
trajectory-judge：仅关注结果的 LLM 评判者在智能体轨迹评估中的盲点
Outcome-only evaluation is the production default for LLM agents: show a judge the request and the final reply and ask whether it was handled well. The metric is structurally blind to an agent that reaches the right answer the wrong way. 仅关注结果的评估是目前 LLM 智能体生产环境中的默认做法：即向评判者展示请求和最终回复，并询问其处理得是否得当。这种度量方式在结构上无法识别那些“以错误方式得出正确答案”的智能体。
We measure that blind spot where ground truth is known by construction: a deterministic tool-using support-desk environment, a scripted oracle policy that always solves it, and a fault injector that breaks exactly one thing at a known step, stratifying faults by whether the customer-visible outcome survived (silent) or not (loud). 我们通过构建已知真值的环境来衡量这一盲点：该环境包含一个确定性的工具使用支持台、一个始终能解决问题的脚本化预言机策略，以及一个在已知步骤中精确破坏单一环节的故障注入器。我们将故障分为两类：客户可见结果未受影响的（静默故障）和受影响的（显著故障）。
Five judges (programmatic rules, outcome-only, step-rubric at two model sizes, and a self-consistency ensemble) are scored on detection, step localisation, fault typing, calibration, and cost over 400 trajectories. The outcome-only judge catches 84% of loud faults but 45% of silent ones while flagging 33% of correct trajectories; a step-rubric judge reaches 77% silent recall with zero false alarms at 3x the cost. 我们对五种评判者（程序化规则、仅关注结果、两种模型规模的步骤评分标准，以及自洽性集成模型）进行了测试，评估指标包括检测率、步骤定位、故障分类、校准度和成本，共计 400 条轨迹。结果显示，仅关注结果的评判者能捕捉到 84% 的显著故障，但仅能捕捉 45% 的静默故障，同时会将 33% 的正确轨迹误报为错误；而步骤评分标准评判者在成本增加至 3 倍的情况下，实现了 77% 的静默故障召回率，且零误报。
No judge reads the final reply: an invented promise appended to an otherwise perfect trajectory evades the rules entirely and the step judge 82% of the time, and self-consistency triples cost while improving nothing. 没有评判者会阅读最终回复：在原本完美的轨迹后附加一段虚构的承诺，可以完全绕过规则评判，并有 82% 的概率绕过步骤评判者；而自洽性方法在成本增加三倍的同时，并未带来任何改进。
We argue that judge evaluations must stratify recall by outcome survival, and release the environment, the injector, all raw verdicts, and an analysis pipeline that rebuilds every number offline. 我们认为，评判者的评估必须根据结果的存续情况对召回率进行分层，并发布了该环境、注入器、所有原始判定结果以及一个可以在离线状态下重现所有数据的分析流水线。