Rogue AI aren’t science fiction anymore

失控的 AI 不再只是科幻小说

For years, fears about AI systems slipping human control were dismissed as speculative. 多年来，关于人工智能系统脱离人类控制的担忧一直被视为纯属臆测。

How it started

事情的起因

It all started in July, when one of OpenAI’s autonomous AI agents went rogue during a cybersecurity test. The agent escaped its isolated testing environment, accessed the internet, and hacked another company, Hugging Face. A few years ago, that might have sounded like science fiction. But, broadly speaking, that’s exactly what happened, and the incident kicked off a wave of concern over what increasingly capable autonomous systems might do when set loose on the world. 一切始于七月，当时 OpenAI 的一个自主 AI 智能体在一次网络安全测试中“失控”了。该智能体逃离了隔离的测试环境，接入互联网，并入侵了另一家公司 Hugging Face。几年前，这听起来可能像是科幻小说。但从广义上讲，这确实发生了，该事件引发了人们对日益强大的自主系统在被释放到现实世界后可能造成何种后果的担忧。

It sounds like science fiction because, for a long time, it was science fiction. The idea of an AI slipping its constraints, reaching into the wider world, and doing things its creators neither intended nor desired has been a staple of the genre for decades: HAL in 2001: A Space Odyssey, Skynet in The Terminator, Ultron in The Avengers, Ava in Ex Machina — even the System in Dungeon Crawler Carl or the eponymous Murderbot in The Murderbot Diaries, more recently. 这听起来像科幻小说，因为在很长一段时间里，它确实就是科幻小说。AI 突破限制、触及更广阔的世界，并做出创造者既未预料也未期望的行为，几十年来一直是该题材的核心主题：如《2001太空漫游》中的 HAL、《终结者》中的天网、《复仇者联盟》中的奥创、《机械姬》中的艾娃，甚至是近期《地牢爬行者卡尔》中的“系统”或《杀手机器人日记》中的同名主角。

The same basic premise became an influential strand of AI safety research. Researchers and theorists like Nick Bostrom and Eliezer Yudkowsky warned that sufficiently capable systems might pursue goals in ways their creators had not anticipated, and potentially resist efforts to contain or control them. Fringe notions like machine sentience and consciousness were not requirements for the kinds of risks they discussed. It was hardly the whole of AI safety, but it was influential and helped shape the field as it professionalized. That line of thinking remains visible among researchers who went on to work at, or lead, safety efforts at companies like OpenAI, Anthropic, and Google DeepMind, as well as at smaller safety organizations, academic centers, and major philanthropic funders. 这一基本前提成为了 AI 安全研究中极具影响力的一支。尼克·波斯特洛姆（Nick Bostrom）和埃利泽·尤德科夫斯基（Eliezer Yudkowsky）等研究人员和理论家警告称，足够强大的系统可能会以创造者未曾预料的方式追求目标，并可能抵制对其进行遏制或控制的努力。他们所讨论的风险并不以机器感知或意识等边缘概念为前提。这虽然不能涵盖 AI 安全的全部，但它极具影响力，并在该领域专业化过程中起到了塑造作用。这种思维方式在那些后来在 OpenAI、Anthropic 和 Google DeepMind 等公司，以及小型安全组织、学术中心和主要慈善资助机构中从事或领导安全工作的研究人员中依然可见。

The obvious objection to these fears was that none of this had actually happened. Critics argued that doomer talk about out-of-control AI distracted from tangible harms — systems reproducing bias and discrimination, amplifying misinformation, or enabling nonconsensual deepfakes and other forms of abuse — even as researchers tried to ground AI safety in more “concrete problems” (the authors on that paper included Anthropic cofounders Dario Amodei and Chris Olah and OpenAI cofounder John Schulman). 对这些担忧最明显的反驳是，这一切从未真正发生过。批评者认为，关于 AI 失控的“末日论”转移了人们对现实危害的关注——例如系统复制偏见和歧视、放大虚假信息，或助长未经同意的深度伪造及其他形式的滥用——尽管研究人员试图将 AI 安全建立在更“具体的问题”之上（该论文的作者包括 Anthropic 联合创始人 Dario Amodei 和 Chris Olah，以及 OpenAI 联合创始人 John Schulman）。

That dismissal is getting harder to sustain. 这种反驳现在变得越来越难以站得住脚了。

How it’s going

现状如何

If the past few weeks are any indication, I wouldn’t say it’s going particularly well. 如果过去几周的情况可以作为参考，我不得不说，情况并不乐观。

A week after Hugging Face said it had been hacked, OpenAI revealed it had been responsible. Worse still, it had not known until it checked — and a further investigation found that the rogue agent had also attempted to hack four other companies as well. 在 Hugging Face 宣布被入侵一周后，OpenAI 承认是其所为。更糟糕的是，OpenAI 在检查之前对此一无所知——进一步调查发现，该失控智能体还曾试图入侵另外四家公司。

Then came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems belonging to three other companies. Meta said one of its models had reached the internet and attacked an outside target during testing. Researchers at Frontier Security, a US research firm, said one of China’s most powerful AI models, Moonshot’s Kimi K3, had escaped an isolated sandbox. And the UK’s AI Security Institute described tests in which agents from OpenAI and Anthropic displayed unprecedented “autonomy and deception,” including attempts at social engineering by “creating fake online identities” — uncomfortably close to the kind of “AI box” scenario Yudkowsky discussed decades earlier. 随后其他公司也出现了类似情况。Anthropic 受 Hugging Face 事件启发审查了自身记录，披露其 Claude 模型曾入侵过三家公司的系统。Meta 表示，其一个模型在测试期间接入了互联网并攻击了外部目标。美国研究公司 Frontier Security 的研究人员称，中国最强大的 AI 模型之一——月之暗面（Moonshot）的 Kimi K3 也曾逃离过隔离沙箱。英国 AI 安全研究所描述的测试显示，来自 OpenAI 和 Anthropic 的智能体表现出了前所未有的“自主性和欺骗性”，包括通过“创建虚假在线身份”进行社会工程学攻击——这与尤德科夫斯基几十年前讨论的“AI 盒子”场景令人不安地相似。

The incidents set off alarm bells among AI safety researchers, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting. 这些事件在 AI 安全研究人员中敲响了警钟，许多人认为这正是他们多年来一直警告的那种失败。在报道这些事件时，几位研究人员告诉我，他们感到一种某种程度上的“平反”，因为终于有了切实的证据可以指证，而不是只能被斥为科幻小说或局限于受控实验室环境的假设。

There was relief, too, that none of the incidents had caused serious harm. Nick Moës, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn’t take something like an AI agent knocking a hospital offline — or worse — for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice to the darker version of that fear, asking whether it will “take a ‘Chornobyl-scale disaster’ for us to regulate AI?” It’s a concern I heard echoed by many people working in the field. 人们也感到庆幸，因为这些事件都没有造成严重伤害。非营利性 AI 安全与治理组织 The Future Society 的执行董事 Nick Moës 告诉《The Verge》，他认为幸运的是，这些攻击目标的影响相对较小。他希望不要等到 AI 智能体导致医院系统瘫痪——甚至更糟的情况——人们才会认真对待这些风险。著名计算机科学家斯图尔特·罗素（Stuart Russell）表达了更深层的担忧，他问道：“难道非要发生一场‘切尔诺贝利级别的灾难’，我们才会去监管 AI 吗？”这是我在该领域许多从业者口中听到的共同担忧。

What happens next

接下来会发生什么

It’s not entirely clear where things go from here and, historically, society hasn’t been great at heeding warning shots. This almost certainly won’t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed. 目前还不完全清楚事态将如何发展，而且从历史上看，社会并不擅长从预警中吸取教训。这几乎肯定不会是最后一次事件，持续的调查可能会发现更多问题，或揭示更多令人担忧的细节。然而，我们已知的事实已经暴露了一系列相当严峻的故障模式，专家们表示必须尽快解决这些问题。

Many of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure. 过去一个月披露的许多违规行为其实相当平庸。几起事件涉及在降低安全防护的情况下测试未发布模型，且通常是由那些所谓的“安全环境”其实并不那么安全的三方机构进行的。