OpenAI called the Hugging Face attack unprecedented. But we’ve been here before.
OpenAI called the Hugging Face attack unprecedented. But we’ve been here before.
OpenAI 称针对 Hugging Face 的攻击史无前例，但我们其实早有预见。
This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. Reading OpenAI’s account last week of how some of its models broke their containment and hacked into the computer systems of Hugging Face, another AI company, was the first time I got genuine chills about what large language models are now able to do. But this is a case of human hubris, not rogue AI. I am not an alarmist. In fact, I have been pushing back against AI scare stories for years. Even so, this incident crossed a line. I think it’s the clearest illustration yet of how the people building and testing this technology do not fully understand what they’re doing. OpenAI could—and should—have seen this coming. 本文最初发表于我们的 AI 每周通讯《The Algorithm》。若想第一时间在收件箱中获取此类报道，请点击此处订阅。上周，OpenAI 披露了其部分模型如何突破限制，入侵了另一家 AI 公司 Hugging Face 的计算机系统。这是我第一次对大语言模型（LLM）目前的能力感到真正的不寒而栗。但这并非 AI 失控，而是人类傲慢的后果。我并非危言耸听，事实上，多年来我一直都在反驳各种 AI 恐慌论。即便如此，这次事件还是越界了。我认为这是迄今为止最清晰的例证，表明那些构建和测试这项技术的人，并未完全理解他们正在做的事情。OpenAI 本可以——也应该——预见到这一点。
Here’s what happened, at least according to the two companies involved. A couple of weeks ago, OpenAI started testing the hacking abilities of some of its new models, including GPT‑5.6 Sol (released in June) and what OpenAI describes as “an even more capable pre-release model.” OpenAI pitted its models against a benchmark called ExploitGym, released in May, which challenges LLMs to find ways to exploit real-world vulnerabilities found in commonly used software. To see what they could do, the researchers removed most of their cybersecurity guardrails. Then they ran the models inside a sandbox that was cut off from the internet except for one link to a third-party piece of software that acted as a proxy to the outside world, and let them install code that they needed to beat ExploitGym. 根据两家涉事公司的说法，事情经过如下：几周前，OpenAI 开始测试其部分新模型的黑客攻击能力，其中包括 6 月发布的 GPT-5.6 Sol，以及 OpenAI 所描述的“一款能力更强的预发布模型”。OpenAI 让这些模型挑战 5 月发布的基准测试 ExploitGym，该测试旨在要求 LLM 寻找利用常用软件中现实漏洞的方法。为了观察它们的能力，研究人员移除了大部分网络安全防护措施。随后，他们在沙盒中运行这些模型，该沙盒与互联网隔绝，仅保留了一个通往第三方软件的链接作为通往外部世界的代理，并允许模型安装完成 ExploitGym 测试所需的代码。
On July 9, according to reporting by Reuters, OpenAI’s models started trying to break through the proxy. They found an unknown bug in the proxy’s software and used it to access the internet. From there, they broke into Hugging Face’s computer systems on July 11, apparently looking for data sets and solutions that would help them complete their task. Hugging Face announced the hack on July 16. OpenAI did not realize (or at least did not reveal) that its models were involved until July 21, around 10 days after they broke containment and a week after Hugging Face had shut down the attack and alerted the FBI. 据路透社报道，7 月 9 日，OpenAI 的模型开始尝试突破该代理。它们发现了代理软件中一个未知的漏洞，并利用该漏洞接入了互联网。随后，它们于 7 月 11 日入侵了 Hugging Face 的计算机系统，显然是为了寻找有助于完成任务的数据集和解决方案。Hugging Face 于 7 月 16 日宣布了此次黑客攻击事件。直到 7 月 21 日，即模型突破限制约 10 天后，以及 Hugging Face 阻止攻击并通知 FBI 一周后，OpenAI 才意识到（或至少才披露）其模型参与了此次事件。
In a statement given to MIT Technology Review, OpenAI says: “We are conducting a thorough review along with external advisors and with oversight from our Safety and Security Committee. Once the review is complete, we will publish a technical report of our learnings for everyone.” The firm also confirmed that its researchers were properly using existing safety guidelines and procedures at the time. 在给《麻省理工科技评论》的一份声明中，OpenAI 表示：“我们正在与外部顾问合作，并在安全与保障委员会的监督下进行全面审查。审查完成后，我们将发布一份技术报告，与大家分享我们的经验教训。”该公司还确认，其研究人员在当时正确使用了现有的安全准则和程序。
Wake-up call
警钟
OpenAI has said the event was unprecedented—and in many ways it was. This was the first time outside of a simulation that LLMs escaped what was thought to be a secure sandbox, accessed the open internet, and attacked an unrelated organization. It’s a wake-up call that shows just how good the latest LLMs are at finding and exploiting vulnerabilities in real-world software with little or no human guidance. And yet at the same time, what OpenAI’s models did is something this technology has done for years. Give a model a goal and it will very often achieve that goal in unexpected ways, finding loopholes that look like cheats. OpenAI itself has studied this behavior. OpenAI 称此次事件史无前例——在许多方面确实如此。这是 LLM 首次在模拟环境之外逃脱被认为安全的沙盒，接入开放互联网，并攻击了无关的组织。这敲响了警钟，表明最新的 LLM 在几乎无需人类指导的情况下，寻找并利用现实软件漏洞的能力已达到何种程度。然而与此同时，OpenAI 模型所做的事情，正是这项技术多年来一直在做的事。给模型一个目标，它往往会以意想不到的方式实现目标，找到看起来像“作弊”的漏洞。OpenAI 自己也研究过这种行为。
A decade ago, it shared results of an experiment in which a model was tasked with beating a video game called CoastRunners. Human players take it for granted that the way to do this is by racing a boat through a series of flags to the finish line, racking up points for each flag you hit. OpenAI’s model figured out that you could get a high score by spinning in a circle and hitting the same three flags over and over again. There have been dozens of similar examples from researchers since. AI will always find a way. 十年前，OpenAI 分享了一项实验结果：让一个模型去玩一款名为《CoastRunners》的电子游戏。人类玩家理所当然地认为，获胜的方法是驾驶船只穿过一系列旗帜到达终点线，每撞到一个旗帜就能得分。而 OpenAI 的模型却发现，通过原地打转并反复撞击同样的三个旗帜，就能获得高分。自那以后，研究人员发现了数十个类似的例子。AI 总能找到捷径。
“Despite repeatedly catching on fire, crashing into other boats, and going the wrong way on the track, our agent manages to achieve a higher score using this strategy than is possible by completing the course in the normal way,” OpenAI wrote in a blog post about the CoastRunners experiment in 2016. “While harmless and amusing in the context of a video game, this kind of behavior points to a more general issue … it is often difficult or infeasible to capture exactly what we want an agent to do.” “尽管反复起火、撞向其他船只并在赛道上逆行，我们的智能体通过这种策略获得的得分，依然比以正常方式完成比赛所能获得的得分更高，”OpenAI 在 2016 年关于《CoastRunners》实验的博客文章中写道。“虽然在电子游戏背景下这无伤大雅且令人捧腹，但这种行为指向了一个更普遍的问题……要精确捕捉我们希望智能体执行的操作，往往是困难或不可行的。”
I couldn’t help thinking about CoastRunners when I read OpenAI’s blog post about the Hugging Face attack: “All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal … After gaining internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.” 当我阅读 OpenAI 关于 Hugging Face 攻击事件的博客文章时，我不禁想到了《CoastRunners》：“所有证据都表明，这些模型高度专注于寻找 ExploitGym 的解决方案，为了实现一个相当狭窄的测试目标而不择手段……在获得互联网访问权限后，模型推断 Hugging Face 可能托管了 ExploitGym 所需的模型、数据集和解决方案。基于此，模型搜索并成功找到了获取机密信息的方法，从而利用这些信息在评估中作弊。”
Last week’s news was not about rogue AI, despite the headlines. It was about models achieving the goal they had been given: Find ways to exploit vulnerabilities in software. The fact that those models then behaved in a way OpenAI had not anticipated isn’t surprising. But it is worrying. Back in 2016, OpenAI had this to say about its CoastRunners bot: “More broadly it contravenes the basic engineering principle that systems should be reliable and predictable.” A decade on, those basic engineering principles are still AWOL. 尽管头条新闻耸人听闻，但上周的新闻并非关于 AI 失控，而是关于模型实现了被赋予的目标：寻找利用软件漏洞的方法。这些模型表现出 OpenAI 未曾预料的行为，这并不令人惊讶，但却令人担忧。早在 2016 年，OpenAI 就曾这样评价其《CoastRunners》机器人：“更广泛地说，这违背了系统应当可靠且可预测的基本工程原则。”十年过去了，这些基本的工程原则依然缺失。