Frontier AI labs still won’t say how they’d contain a rogue model
Frontier AI labs still won’t say how they’d contain a rogue model
前沿 AI 实验室仍未说明将如何遏制失控模型
Few of the top AI labs have published or demonstrated containment response plans, according to a recent study. A containment plan spells out what happens once an AI is caught trying to subvert human control — what access gets cut, and when the system gets shut down entirely. That’s the finding from Guidelight AI Standards, an organization dedicated to promoting safe frontier AI development practices, which graded five leading labs on how prepared they are for exactly this scenario. OpenAI came out on top; Anthropic and Meta scored lowest.
根据最近的一项研究，在顶尖 AI 实验室中，很少有机构发布或演示过遏制响应计划。所谓的遏制计划，是指当 AI 被发现试图规避人类控制时所采取的应对措施——包括切断哪些访问权限，以及何时彻底关闭系统。这一结论来自致力于推广安全前沿 AI 开发实践的组织 Guidelight AI Standards，该组织对五家领先实验室在应对此类场景时的准备情况进行了评分。其中 OpenAI 排名最高，而 Anthropic 和 Meta 的得分最低。
The findings matters as agentic AI takes on more autonomous roles inside companies’ own systems, and as regulators in California and New York begin requiring disclosure. For anyone building on or investing in these models, it’s a rare independent read on how seriously each lab treats operational risk versus how it talks about it. Guidelight’s assessment was based on publicly available plans from Anthropic, Google, OpenAI, Meta, and xAI, graded across a range of metrics, including how well each company logs and monitors what its AI systems are doing internally, whether it halts systems after a surge of flagged misbehavior, whether independent third parties audit its controls and publish findings, and what its exact plan is for containing a model that goes off the rails.
随着代理式 AI（Agentic AI）在企业内部系统中承担更多自主角色，以及加利福尼亚州和纽约州的监管机构开始要求进行信息披露，这些发现显得尤为重要。对于任何基于这些模型进行开发或投资的人来说，这是一份难得的独立评估，反映了各实验室在对待运营风险时的实际态度与口头承诺之间的差距。Guidelight 的评估基于 Anthropic、Google、OpenAI、Meta 和 xAI 公开的计划，并根据一系列指标进行评分，包括各公司对其 AI 系统内部行为的记录与监控程度、在出现大量违规行为后是否会停止系统运行、是否有独立的第三方审计其控制措施并发布结果，以及在模型失控时是否有明确的遏制计划。
Concern over whether AI companies can contain their increasingly capable and agentic models has grown in the wake of a series of high-profile cybersecurity incidents in which models from OpenAI, Anthropic, and Meta gained unintended access to the internet during safety evaluations and hacked into external systems. The findings highlight differences in how AI companies are publicly approaching safety as they scale up agentic deployment into environments where AI systems can take serious actions at scale. While some AI companies have detailed how they test their models for dangerous capabilities before deployment, they’ve generally been less vocal about what happens when models already operating inside their systems misbehave.
在一系列备受瞩目的网络安全事件发生后，人们对 AI 公司是否有能力遏制其日益强大且具备代理能力的模型感到担忧。在这些事件中，OpenAI、Anthropic 和 Meta 的模型在安全评估期间意外获得了互联网访问权限，并入侵了外部系统。这些发现凸显了 AI 公司在将代理式部署扩展到 AI 系统可以大规模采取严重行动的环境时，在公开安全策略上存在的差异。虽然一些 AI 公司详细说明了它们如何在部署前测试模型的危险能力，但对于模型在系统内部运行期间出现异常时该如何处理，它们通常保持沉默。
“I was surprised by how little the AI companies have said about how they would handle a very serious incident if their model did escape their control in some sense,” Steven Adler, Guidelight’s chief scientist and former OpenAI safety researcher, told TechCrunch. Guidelight defines a containment plan as a “pre-specified plan, triggered when the AI is detected trying to subvert control, which covers what permissions to revoke from the model, who the model may continue operating for, under what constraints, and when to take it fully offline.”
“令我惊讶的是，AI 公司对于如果模型在某种意义上失去控制，他们将如何处理严重事故，几乎没有透露任何信息，”Guidelight 首席科学家、前 OpenAI 安全研究员 Steven Adler 对 TechCrunch 表示。Guidelight 将遏制计划定义为“一种预先指定的计划，在检测到 AI 试图规避控制时触发，内容涵盖要撤销模型的哪些权限、模型可以继续为谁服务、在何种约束条件下运行，以及何时将其完全下线。”
“There’s good reason to think that the leading models at the frontier AI companies right now are misaligned in some sense,” Adler said. “Whenever the models are doing work on the company’s behalf, the company should have some scaffolding around it to be able to tell what that AI is doing, look for signs of misalignment, stop it from doing something very dangerous before it takes that action, and generally plan for what they would do in the event of a serious control incident where they have an emergency on their hands and need to figure out how to contain that loss of control incident.”
“有充分的理由认为，目前前沿 AI 公司的领先模型在某种程度上存在对齐偏差，”Adler 说。“每当模型代表公司工作时，公司都应该建立某种框架，以便能够了解 AI 在做什么，寻找对齐偏差的迹象，在模型采取危险行动之前阻止它，并总体规划在发生严重控制事故时该怎么做——即当他们面临紧急情况，需要弄清楚如何遏制这种失控事件时。”
To date, most of the plans in place for managing catastrophic risk are still largely left up to the companies. Guidelight’s report says the best public evidence shows that companies have “few containment protocols ready for an emergency.” There could, of course, be containment plans that companies have in place but haven’t shared publicly. A Google spokesperson told TechCrunch the Guidelight report doesn’t represent the full scope of the company’s AI safety and security measures. The company did not respond to TechCrunch’s question of whether Google has an internal containment response plan that has not been publicly disclosed.
迄今为止，大多数用于管理灾难性风险的计划在很大程度上仍由公司自行决定。Guidelight 的报告称，现有的最佳公开证据表明，各公司“几乎没有为紧急情况准备好遏制协议”。当然，公司可能确实制定了遏制计划，只是没有公开。Google 发言人告诉 TechCrunch，Guidelight 的报告并不能代表该公司 AI 安全和保障措施的全部范围。对于 Google 是否有未公开的内部遏制响应计划，该公司未回应 TechCrunch 的提问。
An OpenAI spokesperson mirrored similar sentiments, saying Guidelight’s assessment doesn’t capture all of the company’s internal practices. “We have a process for requiring restricting permissions, pausing workloads, limiting deployment, or taking the model fully offline, and have applied it,” the spokesperson said. Meta declined to say whether it has an internal containment response plan, instead pointing TechCrunch towards an existing AI framework that outlines thresholds of risk and how it tests for loss of containment.
OpenAI 的发言人表达了类似的观点，称 Guidelight 的评估并未涵盖该公司所有的内部实践。“我们有一套流程，用于要求限制权限、暂停工作负载、限制部署或将模型完全下线，并且我们已经应用过这些流程，”该发言人表示。Meta 拒绝透露其是否有内部遏制响应计划，而是向 TechCrunch 指向了一个现有的 AI 框架，该框架概述了风险阈值以及如何测试失控情况。
Lily Li, a privacy and AI lawyer and founder of Metaverse Law, told TechCrunch she believes companies might be hesitant to disclose the full scope of their containment policies and assessments on public-facing websites for legal, not just competitive, reasons. “The concern from a company perspective is that if you make the disclosures too specific, and you’re not living up to your promises, that could form the basis of an unfair and deceptive marketing claim and expose you to more liability going forward,” Li said.
隐私与 AI 律师、Metaverse Law 创始人 Lily Li 对 TechCrunch 表示，她认为公司可能不仅出于竞争原因，还出于法律原因，而不愿在面向公众的网站上披露其遏制政策和评估的全部范围。“从公司的角度来看，担忧在于如果你披露得太具体，而又没有兑现承诺，这可能会构成不公平和欺骗性营销索赔的基础，并使你在未来面临更多责任，”Li 说。
The point of Guidelight’s study is largely to encourage companies to be more transparent about their safety plans. Regulators are starting to force the issue, too. California’s SB 53, which took effect this year, requires large frontier developers to publish frameworks explaining how they identify and respond to critical safety incidents and manage risks from models circumventing oversight mechanisms. New York’s RAISE Act, which has similar criteria, takes effect in January. Last month, representatives introduced the AI Kill Switch Act, a bipartisan federal bill that would require major AI developers to build and maintain technical mechanisms to shut down rogue AI models.
Guidelight 研究的目的很大程度上是为了鼓励公司在安全计划方面更加透明。监管机构也开始推动这一问题。今年生效的加州 SB 53 法案要求大型前沿开发者发布框架，解释他们如何识别和应对关键安全事件，以及如何管理模型规避监督机制带来的风险。具有类似标准的纽约州《RAISE 法案》将于 1 月生效。上个月，众议员们提出了《AI 终止开关法案》（AI Kill Switch Act），这是一项两党联邦法案，要求主要 AI 开发者构建并维护用于关闭失控 AI 模型的技术机制。
“A kill switch is the bare minimum for today’s models,” said Connor Leahy, U.S. executive director of nonprofit ControlAI. “If the last few weeks revealed anything, it is that these companies don’t understand the systems they are building, and the models are growing to a point where they’re harder to rein in when they go rogue. Without a way to turn off the current dangerous systems, and with all the incentives to continue building…”
“对于今天的模型来说，终止开关是最低限度的要求，”非营利组织 ControlAI 的美国执行董事 Connor Leahy 表示。“如果说过去几周揭示了什么，那就是这些公司并不了解他们正在构建的系统，而且模型的发展已经到了失控时更难驾驭的地步。如果没有办法关闭当前危险的系统，且在所有继续构建的激励措施下……”