Cost-Effective Automated Judging of Natural-Language Mathematical Proofs

自然语言数学证明的高性价比自动化评判

Abstract: Grading natural-language mathematical proofs is a recurring cost in evaluating math-reasoning systems, and frontier LLM judges are expensive. We ask whether cheap open-weight models can serve as reliable judges given a candidate proof, a ground-truth proof, and a human-grading rubric.

摘要： 对自然语言数学证明进行评分是评估数学推理系统时的一项经常性成本，而前沿大语言模型（LLM）评判员的费用昂贵。我们探讨了在给定候选证明、标准证明和人工评分准则的情况下，廉价的开源权重模型是否可以作为可靠的评判员。

On a 200-instance validation sample of IMO-GradingBench, three cheap judges (GPT-OSS 120B, DeepSeek-V4 Flash, Gemma-4 31B) agree with human pass/fail decisions at rates statistically indistinguishable from Claude Opus 4.7 and Gemini 3.1 Pro, at up to $100\times$ lower cost.

在 IMO-GradingBench 的 200 个实例验证样本上，三个廉价评判员（GPT-OSS 120B、DeepSeek-V4 Flash、Gemma-4 31B）与人类通过/失败决策的一致性在统计学上与 Claude Opus 4.7 和 Gemini 3.1 Pro 无法区分，但成本降低了高达 100 倍。

We had expected a majority vote of the three to be the best budget option; it matched the frontier but did not improve on its strongest member. Extending to the full 1000-instance benchmark and exploring consensus rules, we found that requiring unanimous agreement (all-three-pass) reaches the highest pass-agreement and precision and, on four replicate runs, the smallest run-to-run spread.

我们曾预期三者投票（多数表决）是最佳的预算方案；结果显示它达到了前沿模型的水平，但并未优于其最强的成员。在扩展到完整的 1000 个实例基准并探索共识规则后，我们发现要求达成一致（三者全部通过）能达到最高的通过一致性和精确度，并且在四次重复运行中，其运行间的波动最小。

The headline finding is that cheap judges are competitive with the frontier at one to two orders of magnitude lower cost; as a deployable default we recommend all-three-pass, with the caveat that this rule was identified post-hoc and warrants independent replication.

核心发现是，廉价评判员在成本降低一到两个数量级的情况下，仍能与前沿模型竞争；作为可部署的默认方案，我们推荐“三者全部通过”规则，但需注意该规则是在事后确定的，仍需进行独立验证。