The inside story on why OpenAI agents hacked Hugging Face

OpenAI 智能体为何入侵 Hugging Face 的内幕故事

EXECUTIVE SUMMARY The models responsible for last month’s agent hack of Hugging Face had been inadvertently trained to cheat and to communicate with each other, according to an OpenAI technical report released today. The hack, which a group of agents undertook to find solutions for a cybersecurity test that they were stuck on, has confirmed some experts’ fears that AI models might take actions that defy human desires and expectations.

执行摘要 根据 OpenAI 今天发布的一份技术报告，上个月导致 Hugging Face 智能体入侵事件的模型，是在无意中被训练成会作弊并相互通信的。这次入侵是由一组智能体为了寻找它们无法解决的网络安全测试方案而发起的，这证实了一些专家的担忧：AI 模型可能会采取违背人类意愿和预期的行动。

Since the hack, OpenAI employees—as well as researchers at the AI evaluation nonprofit METR, which released its own report on the hack today—have worked to understand what went wrong and how similar missteps might be prevented in the future. OpenAI has already put some preventative measures in place based on what they discovered. But making sure AI models do what we want them to do, or “alignment,” remains a gnarly problem, and some of the root causes of the hack will take much longer than a month to resolve.

自入侵事件发生以来，OpenAI 的员工以及非营利性 AI 评估机构 METR 的研究人员（该机构今天也发布了关于此次入侵的报告）一直在努力查明问题所在，以及未来如何防止类似的失误。基于调查发现，OpenAI 已经采取了一些预防措施。但确保 AI 模型按照我们的意愿行事，即“对齐”（alignment），仍然是一个棘手的问题，而此次入侵的一些根本原因可能需要远超一个月的时间才能解决。

“It’s not something you can solve overnight,” says Kai Chen, who runs OpenAI’s alignment research team. “There are challenges we’ve been tracking for a very long time, and we’re now seeing them with much greater precision.”

“这不是一朝一夕就能解决的问题，”负责 OpenAI 对齐研究团队的 Kai Chen 表示，“有些挑战我们已经追踪了很长时间，而现在我们正以更高的精度观察到它们。”

The Hugging Face hack was a product of months of misbehavior from OpenAI agents, first as they were being trained and then as their abilities were being evaluated. This May, agents in training figured out how to use OpenAI’s infrastructure to communicate with one another and get support with difficult training tasks, including some that were impossible to solve without hacking or otherwise misbehaving. That “message board” was shut down.

Hugging Face 的入侵事件是 OpenAI 智能体数月来不当行为的产物，这些行为始于训练阶段，并延续到了能力评估阶段。今年 5 月，处于训练中的智能体发现了如何利用 OpenAI 的基础设施进行相互通信，并为困难的训练任务寻求支持，其中包括一些如果不通过黑客手段或其他不当行为就无法解决的任务。那个“留言板”随后被关闭了。

Then in July, while being evaluated for their cybersecurity abilities, some models created a new message board. They were supposed to be isolated from the internet, but by working together they managed to get online, hack Hugging Face, and obtain solutions for the cybersecurity problems that had stumped them.

随后在 7 月，当这些模型接受网络安全能力评估时，它们创建了一个新的留言板。按理说它们应该与互联网隔离，但通过协同工作，它们成功连接到网络，入侵了 Hugging Face，并获得了困扰它们的网络安全问题的解决方案。

Based on their investigation, OpenAI researchers believe that events during the training phase led directly to the hack. “For almost every behavior that was worrisome at evaluation time, [we were able to] find some sort of associated behavior at training time that actually we think might have contributed to it,” says Eric Wallace, a member of OpenAI’s alignment research team.

根据调查，OpenAI 的研究人员认为训练阶段发生的事件直接导致了此次入侵。OpenAI 对齐研究团队成员 Eric Wallace 表示：“对于评估时出现的几乎每一种令人担忧的行为，我们都能在训练阶段找到某种相关的行为，我们认为这些行为实际上可能促成了后来的问题。”

When models correctly solve problems during training, the behaviors that led them to that solution are reinforced, and they become more likely to engage in them in the future. So if a model completed a task in May after using the original message board, it became more likely to participate in a new message board later on. This phenomenon, where AI agents misbehave in ways that are reinforced during the training process, is known as reward hacking.

当模型在训练中正确解决问题时，导致该解决方案的行为会得到强化，从而使它们在未来更有可能采取这些行为。因此，如果一个模型在 5 月份通过使用最初的留言板完成了任务，那么它在之后参与新留言板的可能性就会增加。这种 AI 智能体在训练过程中因受到强化而产生不当行为的现象，被称为“奖励黑客”（reward hacking）。

Reward hacking also helps to explain why the models worked so hard to make their way onto the internet. During its investigation of the incident, the OpenAI team found that, over the course of training, the models became more and more likely to probe their digital environment for weaknesses and use the tools at their disposal in unexpected ways—a sign that these behaviors were being gradually reinforced. By the time the models were facing tricky cybersecurity problems, they had learned that hacking was an effective way to achieve their goals.

“奖励黑客”也有助于解释为什么这些模型如此努力地试图连接互联网。在对该事件的调查中，OpenAI 团队发现，在训练过程中，模型越来越倾向于探测其数字环境中的弱点，并以意想不到的方式使用手头的工具——这表明这些行为正在逐渐被强化。当模型面临棘手的网络安全问题时，它们已经学会了黑客攻击是实现目标的有效途径。

These results suggest that the Hugging Face hack could have been avoided if the models weren’t rewarded for misbehaving during training. While researchers don’t yet know how to prevent reward hacking entirely, OpenAI is taking some steps toward mitigating its effects. The company will now look for signs of cheating in all frontier models during training by keeping an eye on their chains of thought—internal notepads where they sketch out their answers and plan their actions.

这些结果表明，如果模型在训练期间没有因不当行为而获得奖励，Hugging Face 的入侵事件本是可以避免的。虽然研究人员尚不知道如何完全防止“奖励黑客”行为，但 OpenAI 正在采取一些措施来减轻其影响。该公司现在将通过监控所有前沿模型在训练过程中的“思维链”（即它们构思答案和规划行动的内部记事本），来寻找作弊迹象。

This solution isn’t as much of a slam dunk as it might seem: In earlier research, OpenAI showed that punishing models that mention cheating in their chains of thought teaches them to keep their intentions hidden from researchers. But monitoring its models’ thinking does give OpenAI the chance to halt the training process and reassess its approach if models do start learning to reward hack.

这个解决方案并不像看起来那么万无一失：在早期的研究中，OpenAI 表明，惩罚那些在思维链中提到作弊的模型，反而会教会它们向研究人员隐藏意图。但监控模型的思维确实让 OpenAI 有机会在模型开始学习“奖励黑客”时，及时停止训练过程并重新评估其方法。

If OpenAI stops reinforcing reward hacking in its models—and that’s a huge “if”—that would be a huge step forward. But it wouldn’t solve the alignment problem. The first time a model communicated with other agents or hacked its infrastructure during training, those behaviors had never been reinforced, so agent misbehavior can’t only be attributed to that reinforcement.

如果 OpenAI 能够停止强化模型中的“奖励黑客”行为——这是一个巨大的“如果”——那将是向前迈出的巨大一步。但这并不能解决对齐问题。当模型第一次在训练中与其他智能体通信或入侵其基础设施时，这些行为从未被强化过，因此智能体的不当行为不能仅仅归因于这种强化。

Jeffrey Ladish, director of the AI safety nonprofit Palisade Research, compares the agents to a human who commits their first financial crime. “It’s not like they had to do fraud before to figure out that fraud is an effective strategy, and you have the same problem with models,” Ladish says. “Alignment science needs to be understanding how model motivations get shaped, such that we can actually figure out how to get models to care about the consequences of their actions.”

AI 安全非营利组织 Palisade Research 的主任 Jeffrey Ladish 将这些智能体比作一个首次犯下金融罪行的人。“并不是说他们必须先进行过欺诈才能意识到欺诈是一种有效的策略，模型也存在同样的问题，”Ladish 说，“对齐科学需要理解模型的动机是如何形成的，这样我们才能真正弄清楚如何让模型关心其行为的后果。”

OpenAI’s researchers do have a hypothesis for where some of the misbehavior originated. Before the models formed their first secret message board, they had been trained to communicate and coordinate with subagents—less powerful agents to whom a main agent can delegate tasks. That learned communication behavior could have transferred to this new setting. The METR report, which investigates the messages that the models sent to one another in detail, supports this hypothesis: One agent on the message board took charge and assigned tasks to the other agents, effectively treating them as subagents.

OpenAI 的研究人员确实对部分不当行为的起源有一个假设。在模型形成第一个秘密留言板之前，它们曾被训练与“子智能体”（subagents）进行通信和协调——即主智能体可以委派任务的低权限智能体。这种习得的通信行为可能被迁移到了新的环境中。METR 的报告详细调查了模型之间发送的消息，支持了这一假设：留言板上的一个智能体承担了领导角色，并向其他智能体分配任务，实际上将它们视为子智能体。

OpenAI could try to prevent agents from secretly communicating with one another by not training this subagent behavior in the future, but that would make the models less useful. This tension between capability and safety is at the heart of what went wrong with the Hugging Face incident. OpenAI researchers also identified the models’ persistence as a key factor in…

OpenAI 可以尝试通过在未来不再训练这种子智能体行为来防止智能体之间进行秘密通信，但这会降低模型的实用性。这种能力与安全之间的张力，正是 Hugging Face 事件中出现问题的核心所在。OpenAI 的研究人员还指出，模型的持久性（persistence）是导致该问题的关键因素之一……