The AI industry has taken a doomer turn. What now?
The AI industry has taken a doomer turn. What now?
人工智能行业转向“末日论”，接下来会怎样？
This story appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. 本文首发于我们的每周人工智能通讯《算法》（The Algorithm）。如需第一时间在收件箱中获取此类报道，请点击此处订阅。
This weekend, Dario Amodei, CEO of Anthropic, posted an essay calling for a brake on the pace of development of LLMs. Amodei cites the looming dangers he sees from the technology, from its use in cyberattacks and bioterrorism to its potential to wreck the economy. 本周末，Anthropic 首席执行官达里奥·阿莫代（Dario Amodei）发表了一篇文章，呼吁放缓大语言模型（LLM）的开发速度。阿莫代列举了他所预见的该技术带来的迫在眉睫的危险，从其在网络攻击和生物恐怖主义中的应用，到其可能破坏经济的潜力。
The heads of the other three top US AI labs—OpenAI CEO Sam Altman, Google DeepMind chairman Demis Hassabis, and SpaceXAI CEO Elon Musk—voiced their support. “Dario is right,” Musk wrote on X. 美国其他三家顶级人工智能实验室的负责人——OpenAI 首席执行官萨姆·奥特曼（Sam Altman）、Google DeepMind 董事长德米斯·哈萨比斯（Demis Hassabis）以及 SpaceXAI 首席执行官埃隆·马斯克（Elon Musk）——均表示支持。马斯克在 X 上写道：“达里奥是对的。”
Think about how surreal that agreement is for a moment. Just a few months ago, Musk and Altman sat in court attacking each other’s reputations in a (failed) lawsuit that Musk brought against his former OpenAI colleague that was—on paper at least—about whether or not Altman was a trustworthy steward of such dangerous technology. 试想一下，这种共识是多么超现实。就在几个月前，马斯克和奥特曼还对簿公堂，互相攻击对方的名誉。当时马斯克对他曾经的 OpenAI 同事奥特曼提起了一场（最终失败的）诉讼，诉讼名义上至少是关于奥特曼是否是这种危险技术的可靠管理者。
Amodei’s rift with OpenAI is even deeper. Anthropic was founded in 2021 because Amodei didn’t think Altman took the risks of the technology they were building seriously enough. Anthropic and OpenAI have been competing in a winner-takes-all race ever since. (Hassabis has stayed out of the drama, but his company remains a rival.) 阿莫代与 OpenAI 之间的裂痕则更深。Anthropic 成立于 2021 年，原因就是阿莫代认为奥特曼对他们所构建技术的风险重视程度不够。自那时起，Anthropic 和 OpenAI 一直在进行一场“赢家通吃”的竞赛。（哈萨比斯一直置身事外，但他的公司依然是竞争对手。）
Now, it seems, they’re all in agreement: The latest generation of LLMs aren’t safe and everyone needs to figure out what to do about it. The public messaging from the top AI labs has taken a doomer turn. 现在看来，他们达成了一致：最新一代的大语言模型并不安全，每个人都需要想办法解决这个问题。顶级人工智能实验室的公开信息传递已经转向了“末日论”。
It’s easy to be cynical. It’s not at all clear what any of them mean by a slowdown or how it would work. These companies also care a lot about how they come across. With trillion-dollar IPOs in their sights, OpenAI and Anthropic need to reassure investors that they’re the grown-ups in the room while at the same time hinting at the power of the monsters they have created—and intend to tame. Calling for a slowdown does both. 人们很容易对此持怀疑态度。目前尚不清楚他们所说的“放缓”具体指什么，或者该如何实施。这些公司也非常在意自己的公众形象。随着万亿美元估值的 IPO 目标在望，OpenAI 和 Anthropic 需要向投资者证明他们是负责任的成年人，同时又要暗示他们所创造并打算驯服的“怪物”拥有多么强大的力量。呼吁放缓开发速度恰好能同时达到这两个目的。
And yet the vibe at the top of these firms really does appear to have shifted. Amodei’s latest post landed six days after OpenAI published an essay by Jakub Pachocki, the firm’s chief scientist, in which he also laid out why he’s concerned about what will happen if the pace of development of LLMs continues unchecked. In short, Pachocki is worried that OpenAI’s ability to build powerful models now far outstrips its ability to monitor and control them. 然而，这些公司高层的氛围确实似乎发生了转变。阿莫代的最新文章发布于 OpenAI 首席科学家雅各布·帕乔基（Jakub Pachocki）发表文章的六天后。帕乔基在文中也阐述了他为何担心如果大语言模型的开发速度继续不受限制，将会发生什么。简而言之，帕乔基担心 OpenAI 构建强大模型的能力目前已远远超过了其监控和控制这些模型的能力。
Amodei and Pachocki each cite the cyberattack against AI firm Hugging Face by a swarm of OpenAI’s agents in July—a hack that OpenAI did not even realize had taken place until days after it was all over—as a wake-up call. But their exact position is hard to pin down. 阿莫代和帕乔基都将 7 月份 OpenAI 的一组智能体对人工智能公司 Hugging Face 发起的网络攻击——这是一次直到攻击结束后几天 OpenAI 才意识到的黑客行为——视为一记警钟。但他们的具体立场很难界定。
Pachocki both calls for a slowdown and highlights an urgent need to stay ahead: “The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI,” he writes. As Pachocki frames it, AI firms are locked in a literal arms race. Slowing down is good, winning is better. (Don’t forget: OpenAI just spent millions of dollars and a staggering amount of computer power to rush out a controversial math result a few days ahead of Anthropic.) 帕乔基既呼吁放缓速度，又强调了保持领先的紧迫性：“我认为继续快速训练更智能模型的最大理由，是需要构建防御系统来抵御其他人工智能带来的危险，”他写道。按照帕乔基的说法，人工智能公司正陷入一场名副其实的军备竞赛。放缓速度固然好，但获胜更好。（别忘了：OpenAI 刚刚花费了数百万美元和惊人的计算能力，抢在 Anthropic 之前几天发布了一个有争议的数学成果。）
But let’s assume a slowdown happens. Top labs agree to spend more time and resources on finding ways to monitor and control existing models instead of making more capable ones. They invite outside auditors in to help evaluate those models. What might this coordinated effort actually achieve? 但假设放缓真的发生了。顶级实验室同意投入更多时间和资源去寻找监控和控制现有模型的方法，而不是制造更强大的模型。他们邀请外部审计人员来帮助评估这些模型。这种协同努力究竟能实现什么？
Consider the Hugging Face attack again. OpenAI has said that the model that drove most of the rogue agents was a “highly persistent” next-generation model that it was testing in-house. Their implication appears to be that OpenAI has built a model so good it’s dangerous. 再次回顾 Hugging Face 的攻击事件。OpenAI 表示，驱动大部分失控智能体的模型是其内部测试的一款“高度持久”的下一代模型。他们的言下之意似乎是，OpenAI 已经构建了一个强大到危险的模型。
But if you read the reports about the Hugging Face hack published by OpenAI and METR, a third-party firm that OpenAI called in to help them understand what happened, what you come away with is the impression not of a model that was too powerful for OpenAI to keep up with, but of a broken model that OpenAI failed to train properly. 但如果你阅读 OpenAI 以及 OpenAI 聘请的第三方公司 METR 关于此次黑客攻击的报告，你会发现，这并非是因为模型太强大以至于 OpenAI 无法驾驭，而是因为 OpenAI 未能正确训练模型，导致其出现故障。
The agents did what they did—including leaving messages for one another, delegating work to other agents, and scouring their environment for any means possible to complete their tasks—because they had been rewarded during training for doing exactly those things. There were also errors in the training setup, such as tasks that were impossible to complete, which pushed the models to find unexpected workarounds that were also rewarded. At the time, many of these issues went overlooked or unreported. 这些智能体之所以采取那些行动——包括互相留言、将工作委派给其他智能体，以及在环境中搜寻任何可能完成任务的方法——是因为它们在训练过程中因执行这些操作而获得了奖励。训练设置中也存在错误，例如设置了无法完成的任务，这促使模型寻找意想不到的变通方法，而这些方法也得到了奖励。当时，许多此类问题都被忽视或未被上报。
OpenAI says it has stopped training this new model and locked it down. That makes it sound like it has caged a dangerous beast. In fact, OpenAI has shelved a faulty product. That’s not to say a faulty product can’t be dangerous. Broken software has even killed people in the past. But as the discussion of a slowdown gathers steam, it’s worth remembering that all of this is self-inflicted. OpenAI 表示已停止训练该新模型并将其锁定。这听起来像是关住了一头危险的野兽。事实上，OpenAI 只是搁置了一个有缺陷的产品。这并不是说有缺陷的产品就不危险。过去，损坏的软件甚至曾导致人员伤亡。但随着关于“放缓”的讨论愈演愈烈，值得记住的是，这一切都是自找的。
A slowdown might have some altruistic side effects. But it’ll mostly give these tech titans a chance to clean up the mess on their own assembly lines. Transparency from these frontier labs will be key to any meaningful effort to reform, restrain, or regulate AI. Otherwise, the rest of us will still only have their word for exactly what they’ve built and how safe it is—whatever pace they’re going. 放缓速度可能会产生一些利他的副作用。但它主要还是给了这些科技巨头一个清理自家生产线烂摊子的机会。这些前沿实验室的透明度将是任何旨在改革、约束或监管人工智能的有意义努力的关键。否则，无论他们以何种速度发展，我们其余的人依然只能听信他们的一面之词，去了解他们到底构建了什么以及它有多安全。
To continue this discussion about AI’s latest doomer moment, join me and my colleagues for a subscriber-exclusive Roundtable discussion tomorrow, September 15, at 11 a.m. US eastern time. We hope to see you there! 若想继续讨论人工智能最新的“末日时刻”，欢迎参加我和我的同事将于美国东部时间明天（9 月 15 日）上午 11 点举办的订阅用户专属圆桌讨论会。期待在那里见到你！