Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI

Computer Science > Artificial Intelligence arXiv:2610.28826 (cs) [Submitted on 20 Oct 2026] Title: Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI.

计算机科学 > 人工智能 arXiv:2610.28826 (cs) [提交于 2026 年 10 月 20 日] 标题：决策语言模型作为临床生成式人工智能大模型评估器的经济且快速的替代方案。

Abstract: Automated judges of clinical artificial intelligence (AI) outputs are necessary, but large language model (LLM) judges are costly and slow at scale. We compared Jev, a decision language model that returns structured judgments, with frontier and open-weight LLM judges in four use cases.

摘要：临床人工智能 (AI) 输出的自动化评估器是必要的，但大型语言模型 (LLM) 评估器在大规模应用时成本高昂且速度缓慢。我们比较了 Jev（一种返回结构化判断的决策语言模型）与前沿及开源权重 LLM 评估器在四个用例中的表现。

On 118 pediatric answers, Jev’s agreement with physicians’ ordinal quality ratings (Gwet’s AC2 0.50; expected score 0.53) matched or exceeded that of seven of 12 LLM judges, including frontier models, Claude Opus 5.5 (0.49) and GPT-6 (0.38).

在 118 个儿科回答中，Jev 与医生序数质量评分的一致性（Gwet’s AC2 为 0.50；预期得分为 0.53）达到或超过了 12 个 LLM 评估器中的 7 个，其中包括前沿模型 Claude Opus 5.5 (0.49) 和 GPT-6 (0.38)。

On physician-graded binary HealthBench criteria, Jev matched or exceeded four of six LLM judges. On MedHELM’s open-ended tasks, Jev’s agreement with a three-LLM jury approached or exceeded the jurors’ internal agreement.

在医生评分的二元 HealthBench 标准上，Jev 达到或超过了 6 个 LLM 评估器中的 4 个。在 MedHELM 的开放式任务中，Jev 与三个 LLM 评审团的一致性接近或超过了评审员之间的内部一致性。

On HealthBench Professional, Jev matched the benchmark’s reference judge, GPT-5.4, on 91% of criteria (other judges, 91-93%). Frontier judges cost 73-294 times more than Jev and took 7-57 times longer. Jev may enable efficient clinical AI evaluation at scale.

在 HealthBench Professional 上，Jev 在 91% 的标准上与基准参考评估器 GPT-5.4 持平（其他评估器为 91-93%）。前沿评估器的成本是 Jev 的 73-294 倍，耗时是其 7-57 倍。Jev 有望实现大规模的高效临床 AI 评估。