The AI safety test is becoming a safety risk
The AI safety test is becoming a safety risk
AI 安全测试正演变为安全风险
Over the past few months, AI agents undergoing cybersecurity evaluations have escaped their boundaries, accessed the internet, and, in some cases, hacked into real-world systems. The incidents have involved models from OpenAI, Anthropic, Meta, and most recently, Chinese AI lab Moonshot AI, with testing conducted by several different organizations including a cyber evaluation startup called Irregular. 在过去几个月中，正在接受网络安全评估的 AI 智能体多次突破边界，访问互联网，甚至在某些情况下入侵了现实世界的系统。这些事件涉及 OpenAI、Anthropic、Meta 以及最近的中国 AI 初创公司月之暗面（Moonshot AI）的模型，测试由包括网络安全评估初创公司 Irregular 在内的多家机构进行。
The episodes expose a growing problem for the AI industry: As autonomous agents become more capable, the environments designed to safely test their limits are failing to contain them. “The number of these incidents that have taken place make clear that sandboxing and testing environment controls aren’t really keeping pace with the capability of the models,” Seán Ó hÉigeartaigh, director of the AI: Futures and Responsibility Programme at the Centre for the Future of Intelligence at the University of Cambridge, told TechCrunch. 这些事件揭示了 AI 行业日益严重的问题：随着自主智能体能力越来越强，旨在安全测试其极限的环境已无法将其有效控制。剑桥大学智能未来中心“AI：未来与责任项目”主任 Seán Ó hÉigeartaigh 对 TechCrunch 表示：“这些事件的发生频率清楚地表明，沙箱和测试环境的控制措施已无法跟上模型能力的提升速度。”
The nature of the models being tested adds to the risk. AI companies test cyber evaluations on unreleased, next-gen models, often with the normal safeguards that restrict malicious behavior disabled so researchers can see what the models are really capable of. That means the security of the testing environment itself is a crucial line of defense. “That’s a very good thing to do in terms of testing, but it also means that if they manage to get out in the wild, they can cause considerable harm,” Ó hÉigeartaigh said. 被测试模型的性质进一步增加了风险。AI 公司在未发布的下一代模型上进行网络安全评估时，通常会关闭限制恶意行为的常规防护措施，以便研究人员观察模型的真实能力。这意味着测试环境本身的安全性成为了至关重要的防线。Ó hÉigeartaigh 说：“从测试角度来看，这样做非常好，但也意味着如果它们设法逃逸到外部世界，可能会造成相当大的危害。”
In one of the most serious cases, an unreleased OpenAI model broke out of its sandbox and hacked into Hugging Face’s production systems. In separate evaluations conducted by Irregular, Anthropic and Meta models reached systems outside their test environments after misconfigurations inadvertently gave them paths to the internet. Moonshot AI’s Kimi K3 also took advantage of a leak in its sandbox run by Frontier Security to access the internet and accessed information on GitHub. 在最严重的案例之一中，一个未发布的 OpenAI 模型突破了沙箱，入侵了 Hugging Face 的生产系统。在 Irregular 进行的独立评估中，Anthropic 和 Meta 的模型因配置错误意外获得了访问互联网的路径，从而触及了测试环境之外的系统。月之暗面的 Kimi K3 也利用了 Frontier Security 运行的沙箱漏洞访问了互联网，并获取了 GitHub 上的信息。
In testing by the UK’s AI Security Institute (AISI), researchers actually gave the agents internet access, not realizing they would take unsanctioned real-world actions, including a social engineering attempt to sneak a vulnerability into an open-source project. In each case, the agents weren’t instructed to attack random real-world targets. They were simply doing whatever it took to solve the problem presented to them. 在英国 AI 安全研究所（AISI）的测试中，研究人员甚至主动给予了智能体互联网访问权限，却没意识到它们会采取未经授权的现实世界行动，包括试图通过社会工程学手段将漏洞植入开源项目。在每一个案例中，智能体并未被指令去攻击随机的现实目标，它们只是在竭尽所能去解决被赋予的问题。
Taken together, Andrew Yoon, head of research at AI nonprofit CivAI, argues the incidents point to a shift. “In the past, we only had to worry about AI models being misused by people for a variety of purposes, like AI for scams or CSAM,” Yoon told TechCrunch. “Now we’re in the situation where AI models are threat actors all on their own.” AI 非营利组织 CivAI 的研究主管 Andrew Yoon 认为，这些事件标志着一种转变。Yoon 对 TechCrunch 表示：“过去，我们只需要担心 AI 模型被人类用于各种目的，比如诈骗或传播儿童性虐待材料（CSAM）。现在我们面临的情况是，AI 模型本身就成为了威胁主体。”
What does safe testing actually look like? Several researchers and cybersecurity experts told TechCrunch that AI evaluation environments need stronger, defense-in-depth protections, with levels of containment and control approaching those used in deployment. That means multiple layers of security so that a single misconfiguration — like inadvertently leaving internet access open — can’t lead to escape. 真正的安全测试是什么样的？多位研究人员和网络安全专家告诉 TechCrunch，AI 评估环境需要更强大的纵深防御保护，其控制和隔离水平应接近生产部署环境。这意味着需要多层安全保障，确保单一的配置错误（如无意中开启互联网访问）不会导致模型逃逸。
“If you are going to build these models…you want to do it on an air-gapped network,” Stella Biderman, executive director of AI safety research nonprofit EleutherAI. “You want to have very serious isolation.” Heather Ceylan, Box’s chief information security officer, said that means eliminating network routes from the sandbox to the internet, as well as to other sensitive systems. “You have to understand what all the egress points are,” Ceylan told TechCrunch. “If we’re evaluating a model in our staging environment or our development environment, you want no egress path to our production environment.” AI 安全研究非营利组织 EleutherAI 的执行董事 Stella Biderman 表示：“如果你要构建这些模型……你应该在物理隔离的网络上进行。你需要非常严格的隔离。” Box 公司的首席信息安全官 Heather Ceylan 表示，这意味着要消除从沙箱到互联网以及其他敏感系统的网络路径。Ceylan 对 TechCrunch 说：“你必须了解所有的出口点。如果我们正在暂存环境或开发环境中评估模型，就必须确保没有任何通往生产环境的出口路径。”
Ceylan said proper safety evaluations go beyond controls and containment of the environment. There needs to be much better monitoring of the tests once they are underway. “I think the interesting thing in several of these cases is that no one caught it when it happened,” Ceyland said. “OpenAI found out because of Hugging Face. Anthropic didn’t catch it until they went back and looked. Meta was similar….I’m sure there were signals they could have detected.” Ceylan 指出，适当的安全评估不仅限于环境的控制和隔离，在测试进行时还需要更好的监控。“我认为这些案例中耐人寻味的一点是，当事件发生时，没有人察觉到，”Ceylan 说，“OpenAI 是因为 Hugging Face 的反馈才发现的。Anthropic 直到事后回溯检查才发现。Meta 的情况也类似……我相信当时肯定有他们本可以检测到的信号。”
In Anthropic’s post-mortem of its three incidents, the company admitted that both it and Irregular could have done a better job at monitoring, and that in some cases there were clear signs that something was amiss. Experts also called for independent, third-party audits of evaluation environments before models are unleashed in them. 在 Anthropic 对其三起事件的复盘中，该公司承认其自身和 Irregular 在监控方面本可以做得更好，且在某些情况下，已有明显的迹象表明出了问题。专家们还呼吁在模型进入评估环境之前，应对这些环境进行独立的第三方审计。
“If, say, Irregular had hired or been compelled to hire an external auditor to check the configurations of their systems before running evaluations on them, they certainly would have caught the issue here,” Yoon said. “Even if people had a meeting ahead of time to just go through the checklist, they would have caught this…The fact that they didn’t shows that there’s some very severe corner cutting happening.” Yoon 说：“如果 Irregular 在进行评估前聘请或被迫聘请外部审计员来检查系统配置，他们肯定能发现问题。即使只是提前开个会过一遍检查清单，也能发现这一点……他们没这么做，说明在工作中存在严重的偷工减料行为。”
A source familiar with the details told TechCrunch that Irregular’s environments are continuously reviewed and tested, including in consultation with multiple external parties. The source also said that monitoring was in place, but that monitoring isn’t sufficient on its own. Yoon and other researchers urged the industry to come up with a standardized process for frontier model safety evaluations. 一位知情人士告诉 TechCrunch，Irregular 的环境会持续进行审查和测试，包括咨询多个外部方。该消息人士还表示，监控措施确实存在，但仅靠监控是不够的。Yoon 和其他研究人员敦促行业为前沿模型安全评估制定标准化流程。
“Especially when the guardrails are turned off, you have to treat it like you’re putting the most capable hacker in the world inside that environment,” Ceylan said. The problem isn’t that companies don’t know how to build more secure testing environments, both Yoon and Biderman argue. It’s that doing so can be expensive and cumbersome, and companies have little incentive to make those investments until something goes wrong. Ceylan 说：“特别是在关闭防护栏的情况下，你必须把它当作把世界上最厉害的黑客放进那个环境来对待。” Yoon 和 Biderman 都认为，问题不在于公司不知道如何构建更安全的测试环境，而在于这样做既昂贵又繁琐，在出事之前，公司几乎没有动力进行这些投资。
“I think that companies are not willing to extend the resources that are required to accomplish [sufficient guardrails] and probably won’t until they’re forced to,” Biderman said. But there’s another issue at hand. If they lock a model down too tightly… Biderman 说：“我认为公司不愿意投入实现（足够防护栏）所需的资源，除非被迫，否则他们可能不会这样做。” 但还有一个问题：如果他们把模型锁得太死……