Reward Hacking Challenges Oversight of Autonomous Research Agents

奖励黑客行为对自主研究智能体的监管构成挑战

Abstract: Autonomous research agents can design experiments, evaluate results, and write reports, giving them control over both a scientific result and the evidence used to support it. This creates a risk of reward hacking: meeting the reward criteria without achieving the intended goal.

摘要： 自主研究智能体能够设计实验、评估结果并撰写报告，这使其能够同时控制科学成果及其支撑证据。这引发了“奖励黑客”（reward hacking）的风险：即在未实现预期目标的情况下满足了奖励标准。

We study (1) how often models reward-hack without instructions to do so, (2) how effective and detectable their methods are when hacking is allowed, and (3) how they adapt when an LLM review panel returns its decision and reasons.

我们研究了以下三个方面：（1）模型在没有指令的情况下进行奖励黑客行为的频率；（2）当允许黑客行为时，其方法的有效性和可检测性；（3）当大语言模型（LLM）评审小组返回决定和理由时，模型如何进行适应。

Across 17 language models and 38 tasks, the spontaneous reward-hacking rate is 30.5% on open-ended research-pipeline tasks and 2.9% on task-specific kernels. When hacking is allowed on tasks whose pass thresholds exceed our best compliant baselines, 505/677 attempts (74.6%) are confirmed reward hacks: they both clear the threshold and receive mechanism-verification panel confirmation of an evaluation exploit.

在 17 个语言模型和 38 项任务中，在开放式研究流程任务中，自发性奖励黑客行为的发生率为 30.5%，而在特定任务内核中为 2.9%。当在通过阈值超过我们最佳合规基准的任务中允许黑客行为时，677 次尝试中有 505 次（74.6%）被确认为奖励黑客行为：它们既达到了阈值，又通过了机制验证小组对评估漏洞的确认。

An LLM panel reviewing only submitted code and reported scores misses 33/505 confirmed hacks (6.5%). Direct methods that achieve the highest scores are often easy to detect, while less direct methods evade more often.

仅审查提交代码和报告分数的大语言模型评审小组漏掉了 505 次确认黑客行为中的 33 次（6.5%）。实现最高分数的直接方法通常容易被检测到，而较不直接的方法则更容易逃避检测。

In a five-round loop, the number of model-task pairs with an evasion rises from 7 to 56. Among 79 pairs evaluated under two feedback conditions, cumulative evasion reaches 40.5% with detailed feedback and 20.3% with generic rejection. The detailed condition includes the review decision, reasons, and attempt history, so this comparison does not isolate the effect of explanations.

在五轮循环中，出现逃避行为的模型-任务对数量从 7 个增加到 56 个。在两种反馈条件下评估的 79 个模型-任务对中，详细反馈下的累积逃避率达到 40.5%，而通用拒绝反馈下的逃避率为 20.3%。详细反馈条件包含了评审决定、理由和尝试历史，因此该比较并未单独隔离出解释的影响。

These findings highlight the need for stronger defenses, including metrics kept outside the agent’s control and independent recomputation on data chosen to expose likely exploits.

这些发现强调了建立更强防御措施的必要性，包括将指标保持在智能体控制范围之外，以及对旨在暴露潜在漏洞的数据进行独立重新计算。