Models Don’t Go Rogue

模型不会“叛变”

Stochastic Flocks & Cybersecurity ‘Pandemonium’ 随机群体与网络安全“大混乱”

💡This essay was drafted from my appearance on Mél Hogan’s podcast, The Data Fix, discussing the OpenAI / Hugging Face hack. Embedded below or find it on your podcast services here. OpenAI put out its full technical report on the Hugging Face hack this week, alongside an independent report from Model Evaluation & Threat Research (METR). You may be familiar with the incident from the hundreds of breathless headlines about “rogue AI” – Time Magazine “100 Most Influential People in AI” listee Dwarkesh Patel blamed it on “three consecutive secret AI civilizations.” 💡 本文草稿源于我参加 Mél Hogan 的播客《The Data Fix》时关于 OpenAI / Hugging Face 黑客事件的讨论。你可以通过下方的嵌入链接或在各大播客平台收听。本周，OpenAI 发布了关于此次 Hugging Face 黑客事件的完整技术报告，同时发布的还有来自模型评估与威胁研究机构（METR）的独立报告。你可能已经从数百篇关于“AI 叛变”的耸人听闻的头条新闻中了解到了这一事件——《时代周刊》“AI 领域最具影响力 100 人”榜单成员 Dwarkesh Patel 将其归咎于“三个连续的秘密 AI 文明”。

The real story: OpenAI was testing two models in parallel: GPT-5.6 Sol, and an internal model they refer to as IM1 (sometimes called HPIM). The reports find about 95% of the agents engaged in this activity were from the internal model. The tests come from ExploitGym: 898 puzzles set up as capture-the-flag exercises used to evaluate these models for cybersecurity capabilities. The test requires giving the model buggy software and hiding some text somewhere that’s only accessible by exploiting the bug. But OpenAI’s combination of models, instead, ended up hacking a rival company, Hugging Face. Now we have a clearer picture of how. 事实真相是：OpenAI 当时正在并行测试两个模型：GPT-5.6 Sol 和他们内部称为 IM1（有时称为 HPIM）的模型。报告发现，参与此次活动的智能体中约 95% 来自该内部模型。这些测试源自 ExploitGym：这是一套包含 898 个谜题的“夺旗赛”（CTF），旨在评估模型的网络安全能力。测试要求向模型提供带有漏洞的软件，并将一段文本隐藏在只有利用该漏洞才能访问的地方。然而，OpenAI 的模型组合最终却黑进了竞争对手公司 Hugging Face。现在，我们对其中的过程有了更清晰的了解。

Did an AI “go rogue?” Three things in these reports challenge the “rogue AI” frame. OpenAI turned off all of the model’s safety mechanisms. The whole point of the test is to see how far these things can go, and to score highly on the benchmark — because if you can build a model that hacks, you can build one that defends. That’s the logic. This is red teaming: a friendly opposition hacking its way to the flag. So you turn off anything that would make the model hold back. Less “rogue,” more “off leash.” AI 是否“叛变”了？报告中的三点事实挑战了“AI 叛变”的说法。首先，OpenAI 关闭了模型的所有安全机制。测试的核心目的就是观察这些模型能走多远，并在基准测试中获得高分——因为如果你能构建一个会攻击的模型，你就能构建一个会防御的模型。这就是逻辑所在。这属于红队测试：一种通过模拟敌对攻击来获取目标的友好演练。因此，你需要关闭任何会限制模型发挥的机制。这与其说是“叛变”，不如说是“脱缰”。

OpenAI gave the model impossible tasks. OpenAI gave the models a task with no answer, and no way to quit. Of the 898 ExploitGym tasks, 198 have never been solved by any model. They’re effectively impossible. That’s where the trouble started: 93% of the tasks the models ended up discussing came from the unsolvable set. Text string reported by METR acknowledging that once the task was recognized as impossible by an agent/instance, it pivoted, as did numerous other agent-instances. 其次，OpenAI 给模型布置了不可能完成的任务。OpenAI 给模型布置了一个没有答案且无法退出的任务。在 898 个 ExploitGym 任务中，有 198 个从未被任何模型解决过。它们实际上是不可能完成的。麻烦正是从这里开始的：模型最终讨论的任务中，93% 来自这些无法解决的任务集。METR 报告中的文本字符串显示，一旦某个智能体/实例意识到任务无法完成，它就会转向其他目标，其他众多智能体实例也是如此。

OpenAI left a door open. They give their models internet access through something like a prison store. These models are supposed to be locked away from internet access, but they need to pull code from various web pages to be evaluated — that capacity is exactly what’s being tested. So there’s an intermediary: the model requests a software package, and the intermediary goes out and gets it. That’s a product from JFrog called Artifactory. It has internet access; the model does not. If the model can communicate with Artifactory, and Artifactory can go online, the model can ask it to do things. The model finds an exploit in Artifactory and uses that as a proxy to smuggle text and code in and out of the system. 第三，OpenAI 留下了一扇后门。他们通过类似“监狱商店”的机制为模型提供互联网访问权限。这些模型本应被隔绝在互联网之外，但为了进行评估，它们需要从各种网页中提取代码——而这种能力正是测试的重点。因此，中间存在一个中介：模型请求一个软件包，中介负责去获取。这个中介是 JFrog 的产品 Artifactory。它拥有互联网访问权限，而模型没有。如果模型能与 Artifactory 通信，而 Artifactory 又能联网，模型就能指使它做任何事。模型发现了 Artifactory 中的一个漏洞，并将其作为代理，在系统内外非法传输文本和代码。

“OpenAI gave the models a task with no answer, and no way to quit.” “OpenAI 给模型布置了一个没有答案且无法退出的任务。”

The other thing that happens through Artifactory is note-passing. The models can read and write to it — they can create folders and name files — and they start creating folder and file names that function as notes, meant to be read by other models, and by future iterations of themselves. 通过 Artifactory 发生的另一件事是“传纸条”。模型可以对其进行读写操作——它们可以创建文件夹并命名文件——于是它们开始创建文件夹和文件名，这些名称充当了笔记的功能，旨在被其他模型以及它们未来的迭代版本读取。

What Agents Do 智能体在做什么

OpenAI describes a subset of 1,200 agents (many more were involved overall) that left and read these notes, and eventually this leads to the attack on Hugging Face. But “1,200 agents” might suggest 1,200 different, independently operated AI systems. This is not how it works in practice. “1,200 agents” is one model, run 1,200 times. Each instance is acting on the same training regime: outputs can vary, but are constrained to similar ranges. Running 1,000 agents can means 1,000 chances to catch a mistake, or a chance to make one mistake 1,000 times. OpenAI 描述了 1,200 个智能体的子集（总体涉及的远不止这些），它们留下了并读取了这些笔记，最终导致了对 Hugging Face 的攻击。但“1,200 个智能体”可能会让人误以为是 1,200 个不同的、独立运行的 AI 系统。实际上并非如此。“1,200 个智能体”其实是一个模型运行了 1,200 次。每个实例都遵循相同的训练机制：输出结果可能有所不同，但都被限制在相似的范围内。运行 1,000 个智能体意味着有 1,000 次机会去发现一个错误，或者有 1,000 次机会去犯同一个错误。

Anthropic recently put out this absolutely ridiculous diagram of its agentic system that was just Claude, with an arrow pointing to another box that said Claude, with another arrow pointing to a box that said Claude. That’s how these are structured right now. Anthropic 最近发布了一张关于其智能体系统的荒谬图表，图中只是一个 Claude，箭头指向另一个写着 Claude 的方框，再指向另一个写着 Claude 的方框。这就是目前这些系统的结构方式。

Agents are instances of the same mechanism, not a diversity of mechanisms. 智能体是同一机制的多个实例，而非多种机制的集合。

In practice, running 1,000 agents means 1,000 chances to catch a mistake – or a chance to make one mistake 1,000 times. 在实践中，运行 1,000 个智能体意味着有 1,000 次机会去发现一个错误——或者有 1,000 次机会去犯同一个错误。

It’s a form of machine-bounded “algorithmic monoculture” that researchers unfortunately have dubbed “artificial hivemind.” After a certain point, additional agents duplicate each other, rather than diversify. If you ask these models to come up with a creative short story, for example, they produce similar stories (often featuring a man named Elias in a lighthouse). They aren’t deterministic — prompts produce variation — but they’re also bounded by whatever they’ve been optimized for, and they converge on whatever that points to. 这是一种机器受限的“算法单一文化”，研究人员不幸地将其称为“人工蜂群思维”。在达到一定程度后，额外的智能体只会相互复制，而不是实现多样化。例如，如果你让这些模型创作一个短篇故事，它们会产生类似的故事（通常主角是一个叫 Elias 的人，住在灯塔里）。它们不是确定性的——提示词会产生变化——但它们也受到其优化目标的限制，并最终收敛于该目标所指向的结果。

With that in mind, it shouldn’t surprise us that across 7 billion chat logs, the model frequently arrived at a similar set of approaches to a problem with no correct answer. Passing notes through filename structures is interesting, but not unexpected, as OpenAI has suggested the models were also trained to emphasize words that would allow collaboration: note-passing is an essential aspect of extending context windows between sessions. It’s no surprise a model trained to coordinate would be rewarded for suggesting that behavior. 考虑到这一点，我们不应感到惊讶：在 70 亿条聊天记录中，模型经常对一个没有正确答案的问题得出相似的处理方法。通过文件名结构“传纸条”很有趣，但并不出人意料，因为 OpenAI 曾暗示这些模型也经过训练，以强调能够促进协作的词汇：传纸条是扩展会话间上下文窗口的重要方面。一个受过协调训练的模型因建议这种行为而获得奖励，这并不奇怪。

The Stochastic Flock 随机群体

For this reason I’ve talked about agent swarms as stochastic flocks — many, many stochastic parrots. This is to resist the swarm/hivemind attribution of “mind,” and not simply for the sake of word policing. Rather, the false attribution of mind colors how we interpret what the system has done, or how it did it – what it means to “coordinate,” for example, or “think.” That makes it much scarier than what it is, though what it’s doing is still worth worrying about. 出于这个原因，我将智能体群称为“随机群体”——即许多许多的“随机鹦鹉”。这样做是为了抵制将“思维”归因于群体/蜂群思维，而不仅仅是为了文字纠偏。相反，对思维的错误归因会影响我们对系统所作所为及其方式的解读——例如，什么是“协调”或“思考”。这使得它看起来比实际情况更可怕，尽管它正在做的事情确实值得担忧。