Anthropic warns that it’s very close to RSI capability, aka self-building AI

While much attention has focused on the updated financials Anthropic revealed in its revised S-1A on Monday, (revenue is soaring, but so are expenses), the AI lab dropped another little bombshell in the fine print.

尽管周一 Anthropic 在修订后的 S-1A 文件中披露的最新财务状况（营收激增，但支出也在同步增长）吸引了大量关注，但这家人工智能实验室在细则中还抛出了另一个重磅炸弹。

The filing seems to warn that the company is on the cusp of creating a model capable of recursive self-improvement (RSI). RSI is the point at which AI grows so good at training itself that it spawns a dramatic acceleration of AI capabilities.

该文件似乎在警告，该公司正处于创造一种具备递归自我改进（RSI）能力模型的边缘。RSI 是指人工智能在自我训练方面变得如此出色，以至于引发人工智能能力急剧加速的临界点。

To the AI doomers, RSI represents the cataclysm from which there is no return, in which AI builds itself into something powerful enough that it slips beyond human control and may be indifferent to the damage it causes while achieving its goals.

对于人工智能末日论者来说，RSI 代表着一场无法回头的灾难，即人工智能将自身构建得足够强大，从而脱离人类控制，并且在实现其目标的过程中可能对所造成的损害漠不关心。

To AI enthusiasts, RSI, or some version of AI reliably using itself to improve, represents the technology’s promise, where it becomes a tool of unimaginable productivity, scientific discovery, and more.

对于人工智能爱好者而言，RSI 或某种可靠地利用自身进行改进的人工智能版本，代表了这项技术的愿景，即它将成为实现难以想象的生产力、科学发现等目标的工具。

Anthropic seems to believe that RSI is both a promise and a peril. It says in the S-1 that it uses AI in its R&D, but it cautions: “Using AI to help build more capable AI could also pose a considerable safety risk, particularly if it sharply accelerates the pace of progress. This is one of the areas covered by our Responsible Scaling Policy, which sets out our plans to monitor and mitigate the risks as we approach what has been termed ‘recursive self-improvement.’”

Anthropic 似乎认为 RSI 既是机遇也是风险。其在 S-1 文件中表示，公司在研发中使用了人工智能，但也警告称：“利用人工智能来帮助构建能力更强的人工智能也可能带来相当大的安全风险，特别是如果它大幅加快了进步速度的话。这是我们《负责任扩展政策》涵盖的领域之一，该政策阐述了我们在接近所谓‘递归自我改进’时监测和缓解风险的计划。”

When the term RSI became popular earlier this year, many believed it was a buzzword or, at best, a goal too hard to achieve, define, or prove. But on Monday, Anthropic added new statements to its Risk Factors section, saying 1) it has already used AI to develop new models whose leaps in capability it didn’t recognize at the time, and 2) it may soon meet its own internal definition of RSI.

当 RSI 一词在今年早些时候流行起来时，许多人认为它只是一个流行语，或者充其量是一个难以实现、定义或证明的目标。但周一，Anthropic 在其“风险因素”部分增加了新的声明，称：1）它已经利用人工智能开发出了新模型，但当时并未意识到这些模型在能力上的飞跃；2）它可能很快就会达到其内部定义的 RSI 标准。

In addition, our increasing use of AI systems to accelerate our own research and development may cause the capabilities of our models to advance more rapidly than our ability to develop, validate, and implement corresponding safety, alignment, and security measures. As a result, we have in the past not identified, and may in the future not identify, a significant acceleration in our rate of development until after it has occurred.

此外，我们越来越多地使用人工智能系统来加速自身的研发，这可能导致我们模型能力的进步速度超过了我们开发、验证和实施相应安全、对齐和保障措施的能力。因此，我们过去未能识别，且未来可能也无法在开发速度出现显著加速时及时察觉，直到加速已经发生。

For example, our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of “dramatically accelerated progress attributable to AI.” We have not designated any of our current models as exceeding this threshold, and we continue to assess the underlying data.

例如，我们对未来模型（而非当前模型）的内部评估和预期表明，我们可能即将达到《负责任扩展政策》中定义的“归因于人工智能的显著加速进步”。我们尚未将任何当前模型指定为超过此阈值，并将继续评估相关基础数据。

However, Anthropic also said in the filing that none of its models have crossed this RSI threshold yet and that it is promising to keep humans in the R&D process for the foreseeable future. “We do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding, if and when that becomes technically possible. Accordingly, we are considering updates to our Responsible Scaling Policy to keep humans in control of frontier model development at Anthropic.”

然而，Anthropic 在文件中也表示，其模型尚未跨越这一 RSI 阈值，并承诺在可预见的未来将人类保留在研发流程中。“即使在技术上成为可能，我们也不认为能够安全地使用 Claude 在没有实质性人类监督或理解的情况下，完全自主地设计和构建后续的前沿模型。因此，我们正在考虑更新《负责任扩展政策》，以确保人类能够掌控 Anthropic 的前沿模型开发。”

Skeptics might point out that an S-1 functions as both disclosure and marketing, and that claims of being close to a model so powerful that it can build itself could fall into the latter category. Nevertheless, there’s no question that Claude’s capabilities, especially in terms of coding, math, and cybersecurity, have grow in leaps and bounds this year. If Anthropic thinks the possibility of RSI is close, maybe we should believe it and act — perhaps govern — accordingly.

怀疑论者可能会指出，S-1 文件既是披露信息也是营销手段，声称接近一种能够自我构建的强大模型可能属于后者。尽管如此，毫无疑问，Claude 的能力，特别是在编程、数学和网络安全方面，今年已经取得了飞跃式的发展。如果 Anthropic 认为 RSI 的可能性已近在咫尺，也许我们应该相信这一点，并据此采取行动——或许是进行监管。