What’s behind the AI industry’s latest warnings of doom?
What’s behind the AI industry’s latest warnings of doom?
AI 行业近期频频发出“末日警告”，背后究竟意欲何为？
The AI industry seems to be having its loudest debate yet about whether its technology poses an existential threat to humanity. The current discussion began after AI researcher Jacob Coxon said that he’s resigned from Anthropic because he’s worried that the leading AI companies are “gambling with our lives.” AI 行业似乎正在经历一场有史以来最激烈的辩论，探讨其技术是否对人类构成了生存威胁。当前的讨论始于 AI 研究员 Jacob Coxon 的表态，他宣布从 Anthropic 辞职，理由是他担心领先的 AI 公司正在“拿我们的生命去赌博”。
Then Anthropic’s alignment lead chimed in with a post declaring, “We really do earnestly believe AI could kill all humans!” adding that he personally thinks the chance is “>10% within the next decade.” On the latest episode of TechCrunch’s Equity podcast, Kirsten Korosec, Sean O’Kane, and I discussed the latest apocalyptic warnings. 随后，Anthropic 的对齐（alignment）负责人发帖称：“我们确实真诚地相信 AI 可能会杀死全人类！”并补充说，他个人认为未来十年内发生这种情况的概率“大于 10%”。在 TechCrunch 的《Equity》播客最新一期节目中，Kirsten Korosec、Sean O’Kane 和我讨论了这些最新的末日警告。
I tried to articulate why I’m skeptical of many AI doomer narratives, while Kirsten asked if this was “just a weird way of flexing to show how far advanced their company’s AI model is,” particularly as these companies prepare to go public. 我试图阐述为什么我对许多 AI“末日论”持怀疑态度，而 Kirsten 则提出质疑：这是否“只是一种展示其公司 AI 模型有多先进的奇怪炫耀方式”，尤其是在这些公司正准备上市的背景下。
And Sean wondered how these concerns might show up in Anthropic’s S-1 filing for its IPO: “Are there junior lawyers right now who are going through and having to rewrite that entire section of the S-1 filing to say, ‘It’s officially Anthropic’s position that there’s a more than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business’?” Sean 则好奇这些担忧将如何体现在 Anthropic 的 IPO S-1 招股书中：“现在是否有初级律师正在逐字审阅，并不得不重写 S-1 文件的整个章节，写道：‘Anthropic 的官方立场是，我们有超过 10% 的概率开发出能够消灭全人类的东西，这对我们的业务将产生重大不利影响’？”
Sean O’Kane: I’m hard-pressed to think of something that blew up so fast. Not only did this warning shot come out from this young researcher who has also worked at OpenAI, but also was immediately shared on X by the alignment lead at Anthropic — who, in what might go down as one of the best misplaced exclamation marks ever, shared Coxon’s post and thread and said, “We really do earnestly believe AI could kill all humans!” Exclamation mark! What a weird vibe. Sean O’Kane： 我很难想到还有什么事情能发酵得这么快。不仅这位曾在 OpenAI 工作过的年轻研究员发出了这枚“警告弹”，而且 Anthropic 的对齐负责人还立即在 X 上转发了它——他可能用了一个史上最不合时宜的感叹号，在分享 Coxon 的帖子和推文串时写道：“我们确实真诚地相信 AI 可能会杀死全人类！”感叹号！这氛围真是太诡异了。
Anthony Ha: Just to disagree with you, I do think that if you believe that AI could destroy all humanity, that does deserve an exclamation point. I would argue that that is a perfectly well used exclamation point! My issue with that tweet was more the “we.” Who is the “we” here? To what extent can we talk about sort of the AI community or AI research community as a monolith? Anthony Ha： 我得反驳你一下，如果你真的相信 AI 会毁灭全人类，那确实值得用一个感叹号。我认为这个感叹号用得恰如其分！我对那条推文的质疑更多在于“我们”这个词。这里的“我们”指的是谁？我们在多大程度上可以将 AI 社区或 AI 研究社区视为一个铁板一块的整体？
Kirsten Korosec: Yeah, I put him in a separate camp than everyone else saying that and talking about the dangers. I’m going to put my speculative hat on, because I want to ask both of you a question, which is: Is it possible that every single time we see the increasing number of blog posts about yet another incident in which one of their AI agents breaks through unintentionally, or they talk about how humanity is at risk, is this a weird way of flexing to show how far advanced their company’s AI model is? Kirsten Korosec： 是的，我把他和那些只会谈论危险的人归为不同阵营。我想发挥一下推测能力，问你们俩一个问题：有没有可能，每当我们看到越来越多的博客文章提到 AI 智能体意外突破限制，或者他们谈论人类面临风险时，这其实是一种展示其公司 AI 模型有多先进的奇怪炫耀方式？
Anthony: I’ve definitely wondered about this. I don’t think it’s completely cynical, in the sense that I don’t think it’s all just a very conscious marketing ploy across the board. I think that when a lot of these people — whether the researchers or CEOs — talk about it, they do have real concern. But of course, it does align with [their] business interests in a lot of ways, to say, “Wow, we’ve built the most deadly software that’s ever been made.” Anthony： 我确实思考过这个问题。我不认为这完全是愤世嫉俗，因为我不觉得这全都是一种刻意的营销手段。我认为当这些人——无论是研究人员还是 CEO——谈论这件事时，他们确实有真实的担忧。但当然，这在很多方面也符合他们的商业利益，即宣称：“哇，我们制造了有史以来最致命的软件。”