Anthropic IPO warns its AI could blackmail people, become sentient, end humanity

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

The Securities and Exchange Commission‘s publication on Monday of Anthropic’s S-1 form—ahead of a heavily anticipated initial public offering—has led to plenty of discussion of whether the company’s quickly ballooning revenues will be able to overcome its even higher costs.

美国证券交易委员会周一公布了 Anthropic 的 S-1 表格，为备受期待的首次公开募股（IPO）做准备。此举引发了大量讨论，人们关注该公司迅速膨胀的收入是否足以抵消其更高的成本。

But amid the numbers and forward-looking optimism about AI “rapidly transform[ing] every sector of the global economy in ways greater than the impact of industrialization, electricity, semiconductors, and the internet,” Anthropic also spends an inordinate amount of time in its S-1 laying out the myriad risk factors that could bring the company down.

然而，在这些数字和关于人工智能“以超过工业化、电力、半导体和互联网影响的方式迅速改变全球经济各个领域”的前瞻性乐观情绪中，Anthropic 也在其 S-1 文件中花费了大量篇幅，列举了可能导致公司倒闭的无数风险因素。

Across 93 pages and over 70,000 words, Anthropic’s S-1 “Risk Factors” section discusses literally hundreds of potential problems that could negatively impact its business in the coming years. Many of these potential risks are the kind of standard worries about competition, costs, customer acquisition, and potential regulation that almost any business must worry about.

在长达 93 页、超过 7 万字的篇幅中，Anthropic 的 S-1“风险因素”部分讨论了数百个可能在未来几年对其业务产生负面影响的潜在问题。其中许多潜在风险属于几乎任何企业都必须担心的标准问题，例如竞争、成本、客户获取和潜在的监管。

But some risks highlight interesting potential points of failure for Anthropic specifically and the hype-filled AI bubble more generally. After wading through Anthropic’s book-length treatise on its greatest risks, we’ve narrowed the list to the seven most surprising, interesting, or thought-provoking potential problems.

但有些风险凸显了 Anthropic 特有的，以及更广泛的、充满炒作的人工智能泡沫中一些有趣的潜在故障点。在研读了 Anthropic 这份长篇大论的风险报告后，我们将名单缩小到了七个最令人惊讶、最有趣或最发人深省的潜在问题。

An existential crisis: Let’s start with the biggest potential risks: Those that could “pose catastrophic or existential risks to humanity,” as Anthropic writes. The company acknowledges that concerns about these risks “have been raised both within our organization and by other researchers, regulators, policymakers, and industry participants.”

生存危机：让我们从最大的潜在风险开始：正如 Anthropic 所写，那些可能“对人类构成灾难性或生存风险”的因素。该公司承认，关于这些风险的担忧“无论是在我们组织内部，还是在其他研究人员、监管机构、政策制定者和行业参与者中，都已被提出”。

The specific worry isn’t in any current Anthropic model, the company says, but near-future models that could “substantially automate AI research and development” on their own. If these systems were “allowed to operate with a high degree of autonomy,” they could quickly advance to a point that “accelerate[s] AI progress beyond our ability to evaluate and safeguard new model,” Anthropic warns.

该公司表示，具体的担忧并非针对当前的任何 Anthropic 模型，而是针对不久的将来可能“自主实现人工智能研发自动化”的模型。Anthropic 警告称，如果这些系统“被允许以高度自治的方式运行”，它们可能会迅速发展到“加速人工智能进步，超出我们评估和保护新模型能力”的地步。

Somewhat hilariously, Anthropic writes that, besides threatening all of humanity, this kind of self-improving AI “could adversely affect our business, financial condition, and results of operations.” If the worst comes to pass, as some prominent critics have warned, we don’t think Anthropic’s business travails will rank near the top of humanity’s concerns.

颇为滑稽的是，Anthropic 写道，除了威胁全人类之外，这种自我改进的人工智能“还可能对我们的业务、财务状况和经营业绩产生不利影响”。如果最坏的情况发生，正如一些著名批评家所警告的那样，我们认为 Anthropic 的业务困境将不会是人类最关心的问题。

It’s alive! Most IPOs don’t have to address the concern that their products “possess or could develop forms of sentience or moral status.” But Anthropic is among that vanishingly small minority that has to analyze what might happen if governments and society at large start treating Claude less like a computer program and more like a person.

它活了！大多数 IPO 不需要解决其产品“拥有或可能发展出某种感知能力或道德地位”的担忧。但 Anthropic 属于极少数需要分析以下情况的公司：如果政府和社会大众开始不再把 Claude 仅仅当作一个计算机程序，而是当作一个人来对待，会发生什么。

This risk isn’t surprising, given that Anthropic has been relatively upfront about the AI sentience debate playing out inside its walls. If those debates lead to widespread acknowledgment that AI is sentient, Anthropic notes that it could lead to “regulatory requirements, public pressure, or advocacy efforts” that could constrain the company’s operations.

考虑到 Anthropic 对其内部关于人工智能感知能力的辩论一直相对坦诚，这一风险并不令人意外。Anthropic 指出，如果这些辩论导致人们普遍承认人工智能具有感知能力，那么这可能会引发“监管要求、公众压力或倡导活动”，从而限制公司的运营。

Short of that, the company itself could “elect to adopt policies, safeguards or limitations … that go beyond existing legal requirements” if it thinks such actions are necessary. The sterile language undersells the seismic cultural shift that would result from any widespread acknowledgment of AI sentience.

即便不至于此，如果公司认为有必要，它也可能“选择采取超出现有法律要求的政策、保障措施或限制”。这种枯燥的语言掩盖了任何对人工智能感知能力的广泛认可所带来的巨大文化变革。

Who’s controlling who? Continuing the theme of “IPO risks that sound like they come from a science fiction novel,” Anthropic spends a good deal of time discussing the possibility that its models could display “unexpected strategic or self-preserving behaviors” to prevent control from human operators. That includes “outputs that could be interpreted as coercive, deceptive, or manipulative, including behavior resembling blackmail.”

谁在控制谁？延续“听起来像是科幻小说中的 IPO 风险”这一主题，Anthropic 花了大量时间讨论其模型可能表现出“意想不到的战略或自我保护行为”，以防止人类操作员控制的可能性。这包括“可能被解读为强制性、欺骗性或操纵性的输出，包括类似勒索的行为”。

If the “AI blackmail” thing sounds familiar, you may remember a story from last year in which OpenAI models edited shutdown scripts and threatened engineers in highly contrived and controlled scenarios. As we wrote at the time, “the danger isn’t AI systems sprouting intentions—it’s that we’ve created systems that can manipulate human psychology through language.

如果“人工智能勒索”听起来很熟悉，你可能还记得去年的一个故事，其中 OpenAI 的模型在高度设计和受控的场景中编辑了关闭脚本并威胁了工程师。正如我们当时所写，“危险不在于人工智能系统萌生了意图，而在于我们创造了可以通过语言操纵人类心理的系统。”

These days, Anthropic notes that perceptions of manipulation by AI models “could generate significant public concern, regulatory scrutiny, litigation risk, or reputational harm, regardless of whether such behavior results in actual harm.” What Anthropic doesn’t address in its S-1 is the possibility that AI models could manipulate us all in ways we don’t even realize.

如今，Anthropic 指出，人们对人工智能模型操纵的看法“可能会引发重大的公众担忧、监管审查、诉讼风险或声誉损害，无论这种行为是否造成了实际伤害”。Anthropic 在其 S-1 文件中没有提到的是，人工智能模型可能以我们甚至没有意识到的方式操纵我们所有人的可能性。

Give a damn ’bout my bad reputation: Throughout the S-1’s Risk Factors section, Anthropic mostly focuses on problems caused by its own AI models or business moves. But the company also acknowledges that problems with AI’s broader reputation could pose a threat, even if Anthropic does nothing wrong. At one point, Anthropic imagines a hypothetical in which…

不在乎我的坏名声：在整个 S-1 的“风险因素”部分，Anthropic 主要关注由其自身人工智能模型或商业举措引起的问题。但该公司也承认，即使 Anthropic 本身没有做错任何事，人工智能更广泛的声誉问题也可能构成威胁。在某一处，Anthropic 设想了一个假设情况，即……