GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

GAUGE：在任务导向型智能体用户模拟评估中，何时不应信任“大模型作为裁判”

Comparing and selecting task-oriented LLM agents increasingly relies on a low-cost offline evaluation gate: persona-driven LLM user-simulators converse with each candidate, an LLM-as-a-judge scores the transcripts, and the higher-scoring agent is promoted. 在比较和选择任务导向型大模型（LLM）智能体时，人们越来越依赖一种低成本的离线评估门控机制：即通过角色驱动的大模型用户模拟器与每个候选智能体进行对话，由“大模型作为裁判”（LLM-as-a-judge）对对话记录进行评分，并选拔出得分较高的智能体。

We introduce GAUGE, a reusable offline protocol that measures whether this gate’s ranking matches a grounded verifiable reward across 25 agents from six providers on the $\tau^2$-bench and SimulatorArena benchmarks, separating two kinds of evaluation validity that release practices conflate: ranking validity and construct validity. 我们引入了 GAUGE，这是一个可重复使用的离线协议。该协议通过 $\tau^2$-bench 和 SimulatorArena 基准测试，评估了来自六家供应商的 25 个智能体，旨在衡量该门控机制的排名是否与基于事实的可验证奖励相符。此外，该协议还区分了当前发布实践中常被混淆的两种评估有效性：排名有效性（ranking validity）和构念有效性（construct validity）。

First, a satisfaction-success gap: satisfaction carries essentially no information about task success, as conversations rated satisfied by our blind panel are decorrelated from actual success, with 57.5% of them failing the customer’s task, a pattern consistent across five rater populations, both benchmarks, and every subjective dimension we rated. 首先是“满意度与成功率之间的鸿沟”：满意度评分几乎不包含关于任务成功与否的信息。我们的盲测小组认为“满意”的对话与实际任务成功之间并无关联，其中 57.5% 的对话实际上未能完成客户的任务。这一模式在五个不同的评估群体、两个基准测试以及我们评估的每一个主观维度中均保持一致。

Second, while the gate’s ranking is robust across the broad capability span, it loses resolution among the near-equal strong agents: this decision-disagreement rate jumps from $<$1% on wide-reward pairs to 31% on close pairs. The gate is thus human-validated yet mis-anchored. 其次，虽然该门控机制在广泛的能力范围内表现稳健，但在能力相近的强智能体之间却失去了分辨力：在奖励差距较大的智能体对中，决策分歧率低于 1%，但在能力接近的智能体对中，这一比例跃升至 31%。因此，该门控机制虽然经过了人工验证，但其基准锚定存在偏差。

As a remedy, we propose a calibrate-then-trust cadence in which a judge-free completion bit is a zero-cost tripwire for truncation regressions. 作为补救措施，我们提出了一种“先校准、后信任”的节奏，即利用无需裁判的“完成位”（completion bit）作为一种零成本的触发机制，用于监测任务截断回归问题。