Evaluating Model Capabilities After Benchmark Saturation

Computer Science > Artificial Intelligence arXiv:2610.20667 (cs) [Submitted on 14 Oct 2026] Title: Evaluating Model Capabilities After Benchmark Saturation Authors: Morio Hamada, Jehyeok Yeon, Matteo Migliarini, Justin Olive, Magda Dubois.

计算机科学 > 人工智能 arXiv:2610.20667 (cs) [提交于 2026 年 10 月 14 日] 标题：基准饱和后的模型能力评估 作者：Morio Hamada, Jehyeok Yeon, Matteo Migliarini, Justin Olive, Magda Dubois。

Abstract: Rapid advances in AI capabilities are causing benchmarks to saturate. Once several models approach perfect scores, a benchmark loses its power to differentiate between them, and with it its utility for informing AI safety cases and frontier policies. Yet benchmark items produce far more information than the scores we typically extract from them: a full transcript of how each model solved the task.

摘要：人工智能能力的快速进步正导致基准测试趋于饱和。一旦多个模型接近满分，基准测试就失去了区分它们的能力，同时也失去了为人工智能安全案例和前沿政策提供参考的效用。然而，基准测试项目所产生的信息远不止我们通常从中提取的分数：它还包含模型如何解决任务的完整记录。

We investigate whether these transcripts can be used to assess model capabilities even after benchmark scores have saturated. We simulate saturation retrospectively using historical benchmark and model releases, and test whether protocols that access only correctly solved transcripts from saturated benchmarks can predict capabilities measured on held-out, unsaturated items.

我们研究了即使在基准分数饱和后，这些记录是否仍可用于评估模型能力。我们通过历史基准和模型发布回顾性地模拟了饱和状态，并测试了仅访问饱和基准测试中正确解决的记录的协议，是否能够预测在未公开的、未饱和项目上测得的能力。

We evaluate representative contemporary approaches for extracting information from model transcripts: pairwise judging, single-transcript absolute scoring, and predictions from deterministic transcript features or embeddings. We find that saturated transcripts retain capability information beyond benchmark accuracy.

我们评估了从模型记录中提取信息的代表性当代方法：成对判断、单记录绝对评分，以及基于确定性记录特征或嵌入的预测。我们发现，饱和的记录保留了超出基准准确率的能力信息。

Amongst our evaluated protocols, pairwise judging performs best, reducing prediction error by 5.3 times compared to a baseline using only saturated benchmark scores. This capability-relevant information is distributed across transcript sections, but concentrated in specific channels of information such as reasoning and tool-calls.

在我们评估的协议中，成对判断表现最佳，与仅使用饱和基准分数的基线相比，预测误差降低了 5.3 倍。这种与能力相关的信息分布在记录的各个部分，但集中在推理和工具调用等特定的信息渠道中。

However, the predictive power of protocols does not always generalise across distribution shifts — for example by underestimating the capabilities of the first reasoning models introduced in early 2025. Saturated benchmarks thus retain capability-predicting signal in their task transcripts, but extending the lifetime of existing benchmarks requires methods designed to extract these signals robustly.

然而，协议的预测能力并不总能在分布偏移中保持通用——例如，它低估了 2025 年初推出的首批推理模型的能力。因此，饱和的基准测试在其任务记录中保留了预测能力的信号，但要延长现有基准测试的寿命，需要设计能够稳健提取这些信号的方法。