EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models

EvalDetectBench：衡量前沿语言模型“评估意识”的基准测试

Frontier large language models can often recognize when they are being evaluated, a capability known as evaluation awareness. If models behave differently in evaluations than in deployment, this undermines the validity of evaluation results, which are a crucial component of current AI safety frameworks.

前沿大语言模型通常能够识别出它们何时正在接受评估，这种能力被称为“评估意识”（evaluation awareness）。如果模型在评估中的表现与实际部署时不同，这将削弱评估结果的有效性，而评估结果正是当前人工智能安全框架的关键组成部分。

We introduce EvalDetectBench, an open pipeline and benchmark for measuring evaluation awareness that works with any Inspect-compatible evaluation, allowing practitioners to test against current and future benchmarks. EvalDetectBench ships with a newly curated transcript suite covering current frontier system-card evaluations and diverse deployment sources.

我们推出了 EvalDetectBench，这是一个用于衡量评估意识的开放式流水线和基准测试工具。它适用于任何兼容 Inspect 的评估，允许从业者针对当前及未来的基准进行测试。EvalDetectBench 附带了一套新策划的转录数据集，涵盖了当前前沿系统的评估报告以及多种部署来源。

The benchmark serves two purposes: measuring how reliably frontier LLMs recognize that they are being evaluated, and assessing how detectable individual benchmarks are as evaluations.

该基准测试有两个目的：一是衡量前沿大语言模型识别其正在接受评估的可靠性；二是评估各个基准测试作为“评估行为”本身的可检测性。

We identify two methodological choices in the existing literature that introduce systematic bias: the identity of the model that generated the deployment transcripts accounts for 11.25% of measurement variance and can reorder model rankings; and elicitation prompts selected for high performance on one model can perform near chance on others. EvalDetectBench corrects for both via per-model probe calibration and a stratified generator-harmonisation procedure.

我们发现现有文献中存在两种引入系统性偏差的方法论选择：首先，生成部署转录的模型身份占测量方差的 11.25%，并可能导致模型排名发生变动；其次，为在某一模型上获得高性能而选择的诱导提示词（elicitation prompts），在其他模型上可能表现得如同随机猜测。EvalDetectBench 通过针对每个模型的探测校准（probe calibration）和分层生成器协调程序（stratified generator-harmonisation procedure）纠正了这两个问题。