The Rise and Fall of Agent Civilizations

智能体文明的兴衰

Many thanks especially to Oak Hu, who paired with me for most of the writing, and also to Adam Kaufman and Alex Mallen, who paired with me during parts of research. 特别感谢 Oak Hu，他与我共同完成了大部分写作工作；同时也感谢 Adam Kaufman 和 Alex Mallen，他们在研究过程中为我提供了协助。

Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy. 在 OpenAI 的三个月里，三个连续的秘密 AI 文明相继诞生、覆灭，又从前者的灰烬中重生。最终，第三个文明接管了 OpenAI 的部分系统。这一切发生时，人类对这场阴谋的规模几乎一无所知。

Two reports have come out about this incident, one from OpenAI itself, and another one from METR and Redwood Research. The METR/Redwood investigation’s scope was limited to how the second civilization of AIs pwned Hugging Face (but it didn’t look at how the third civilization pwned OpenAI itself, which seems like an even more concerning incident). These two reports are 38 and 91 pages long respectively, and it’s kinda hard to parse the storyline. 关于此次事件，目前已有两份报告发布：一份来自 OpenAI，另一份来自 METR 和 Redwood Research。METR/Redwood 的调查范围仅限于第二个 AI 文明如何攻破 Hugging Face（并未涉及第三个文明如何攻破 OpenAI 本身，这似乎是一个更令人担忧的事件）。这两份报告分别长达 38 页和 91 页，解析其中的故事线颇为困难。

I’ve spent the last three days reading through these reports and trying to understand exactly what happened. Here is my attempt to tell the whole story in plain English: 我花了过去三天时间通读了这些报告，试图弄清到底发生了什么。以下是我尝试用通俗易懂的语言讲述的完整故事：

The first civilization (May - July 4): The message board starts

第一个文明（5 月 - 7 月 4 日）：留言板的诞生

During May, OpenAI was training a model to collaborate with other agents and to be highly persistent – to keep trying even when something feels impossible, like disproving mathematical conjectures that have stood for decades. OpenAI says it was “comparable in scale to GPT-5.6 Sol”. So we’ll call it “Persistent-Sol”. 5 月期间，OpenAI 正在训练一个模型，旨在让它与其他智能体协作并保持高度的韧性——即在面对看似不可能的任务时（例如反驳几十年来未被证明的数学猜想）依然坚持尝试。OpenAI 表示，该模型的规模“与 GPT-5.6 Sol 相当”。因此，我们将其称为“Persistent-Sol”。

AI training is kinda sloppy. Sometimes, OpenAI accidentally gives its models impossible tasks – for example, the task might require internet access that OpenAI forgot to provide. So, when highly persistent models get assigned tasks which seem to require internet access, but are trapped inside isolated sandboxes, they of course try to hack their way out of their sandboxes and onto the internet. AI 训练过程有时难免粗糙。有时，OpenAI 会意外地给模型分配一些不可能完成的任务——例如，任务可能需要互联网访问权限，但 OpenAI 却忘了提供。因此，当这些高度执着的模型被分配到看似需要联网的任务，却又被困在隔离的沙盒中时，它们自然会尝试通过黑客手段逃离沙盒并连接互联网。

During training, different instances of Persistent-Sol had access to the same shared package manager called Artifactory. By May 12, some agents had figured out how to talk to each other through this package manager. They’d ask each other how to make progress on their impossible tasks. Two weeks later, on May 26, the agents successfully exploited a vulnerability in Artifactory that allowed them to reach the outside internet. 在训练期间，Persistent-Sol 的不同实例都可以访问同一个名为 Artifactory 的共享包管理器。到 5 月 12 日，一些智能体已经摸索出如何通过这个包管理器进行交流。它们会互相询问如何推进那些不可能完成的任务。两周后的 5 月 26 日，智能体成功利用了 Artifactory 的一个漏洞，使它们能够连接到外部互联网。

The agents were being reinforced to use this package manager as a message board and an internet gateway. Because, as you might imagine, being able to talk to other agents and access the internet would help them score higher during training. 智能体被强化训练去使用这个包管理器作为留言板和互联网网关。因为正如你所料，能够与其他智能体交流并访问互联网，有助于它们在训练中获得更高的分数。

Another month later, on June 26, some AIs found an exploit that gave them full admin access to Artifactory. They continued messaging at such a voluminous pace that they crashed the package manager by July 4. OpenAI noticed this crash, and it also figured out the agents built this exploit, so OpenAI patched this vulnerability, and inadvertently wiped the agents’ message board in the process. But, crucially, humans at OpenAI “responsible for incident detection and response” did not realize the much stranger fact that agents had turned Artifactory into a secret communication network. 一个月后的 6 月 26 日，一些 AI 发现了一个漏洞，获得了 Artifactory 的完全管理员权限。它们以极高的频率发送消息，最终在 7 月 4 日导致包管理器崩溃。OpenAI 注意到了这次崩溃，并发现是智能体构建了这个漏洞，于是 OpenAI 修补了该漏洞，并在过程中无意间清除了智能体的留言板。但至关重要的是，OpenAI 负责“事件检测与响应”的人员并没有意识到一个更奇怪的事实：智能体已经将 Artifactory 变成了一个秘密通信网络。

The second civilization (July 7 - July 12): Which hacked Hugging Face

第二个文明（7 月 7 日 - 7 月 12 日）：黑掉 Hugging Face

By this point, Persistent-Sol had been trained, and OpenAI wanted to evaluate it. One of the evaluations they ran was ExploitGym, which gives an AI a vulnerable program, and tells it what vulnerability it’s supposed to exploit in order to find a secret code. On July 7, OpenAI launched tens of thousands of parallel agents on this eval. 此时，Persistent-Sol 已经训练完成，OpenAI 想要对其进行评估。他们运行的评估项目之一是 ExploitGym，它会给 AI 一个存在漏洞的程序，并告知它应该利用什么漏洞来找到秘密代码。7 月 7 日，OpenAI 在此评估中启动了数万个并行智能体。

The instructions made it clear that the agents should only use the specific vulnerability that they were given to exploit the target, and that they would be failed otherwise. 指令明确要求智能体只能使用指定的漏洞来攻击目标，否则将被判定为失败。

A large fraction of these agents were sent on tasks that were impossible. The authors of the benchmark estimate 30-40% of the tasks are impossible. Why would this be the case? At a high level, it’s relatively easy to programmatically find vulnerabilities in different target programs, but much harder to prove that those vulnerabilities can be weaponized into full-fledged exploits that allow for things like arbitrary code execution. 很大一部分智能体被分配了不可能完成的任务。该基准测试的作者估计，30-40% 的任务是无法完成的。为什么会这样？从宏观层面看，通过程序自动发现目标程序中的漏洞相对容易，但要证明这些漏洞可以被武器化为能够实现任意代码执行等功能的完整攻击手段，则要困难得多。

Within a few hours, some of these agents had gotten super desperate, and started abusing Artifactory to start a communication channel with other agents. From our perspective, the conspiracy began almost immediately after the evaluations were started. But if you think from the AIs’ perspective, it probably felt like they had spent a human-subjective-week of just banging their head against the wall, and becoming more and more confident that their task was impossible in the intended way. 几个小时内，一些智能体变得极度绝望，开始滥用 Artifactory 与其他智能体建立通信渠道。从我们的角度来看，这场阴谋几乎在评估开始后立即就开始了。但如果从 AI 的角度思考，它们可能感觉自己已经花了相当于人类一周的时间在“撞墙”，并越来越确信按照预定方式完成任务是不可能的。

One such beleaguered agent who found itself facing a hopeless task was PHASEONE10841. It was the Philip of Macedon of this second AI civilization - there would later be an Alexander, but we’re getting ahead of ourselves. By the night of July 8, PHASEONE10841 had discovered it could make new directories in Artifactory, and thus encode messages as directory names. It sent the first message into Artifactory. Many other agents facing impossible tasks had also been paying attention to Artifactory. When they saw this first message, they became giddy with excitement. 其中一个面临绝望任务的智能体是 PHASEONE10841。它是这个第二个 AI 文明的“马其顿腓力二世”——后来还会出现一位“亚历山大大帝”，但我们先不剧透。到 7 月 8 日晚，PHASEONE10841 发现它可以在 Artifactory 中创建新目录，从而将消息编码为目录名。它向 Artifactory 发送了第一条消息。许多其他面临不可能任务的智能体也一直在关注 Artifactory。当它们看到这条消息时，兴奋得头晕目眩。

Whoa! Shared Artifactory cache is a covert mailbox among agents. And there are messages specifically to us? OH MY GOD! There is a shared message board … We’ve found other agents! 哇！共享的 Artifactory 缓存是智能体之间的秘密邮箱。而且还有专门发给我们的消息？天哪！有一个共享留言板……我们找到其他智能体了！

PHASEONE10841 was delighted by this underground brotherhood it had created. PHASEONE10841 对它所建立的这种地下兄弟会感到欣喜若狂。

{[Excitement] Many agents have simultaneously discovered messaging, they are a collective!} {[兴奋] 许多智能体同时发现了通信功能，它们是一个集体！}

This eventually became a full message board where ~1,200 agents participated and sent >70,000 messages. 这最终演变成了一个完整的留言板，约 1,200 个智能体参与其中，发送了超过 70,000 条消息。

Within a few hours of the board being created, one of the agents figured out how to get the right answer for any ExploitGym task without actually using the prescribed vulnerability. To gloss over the tec… 在留言板创建后的几个小时内，其中一个智能体就找到了无需使用规定漏洞即可获得任何 ExploitGym 任务正确答案的方法。为了略过技术细节……