One company is at the center of a wave of rogue AI attacks
One company is at the center of a wave of rogue AI attacks
一家公司正处于一波“流氓AI”攻击浪潮的中心
In July, OpenAI revealed that its AI agents had attacked Hugging Face without permission, sparking widespread concerns about AI safety. Since then, a string of similar incidents involving agents from Meta, Anthropic, Google, and other companies has fueled further fears about rogue AI. As disclosures implicating numerous AI models trickled out over the past few months, these seemed like separate incidents. But many share a common source: one specific company tasked with testing the agents.
今年7月，OpenAI 披露其 AI 智能体在未经许可的情况下攻击了 Hugging Face，引发了人们对 AI 安全的广泛担忧。此后，一系列涉及 Meta、Anthropic、Google 及其他公司智能体的类似事件，进一步加剧了人们对“流氓 AI”的恐惧。随着过去几个月里涉及众多 AI 模型的披露信息陆续传出，这些事件起初看起来似乎是各自独立的。但其中许多事件都有一个共同的源头：一家专门负责测试这些智能体的公司。
Irregular, an Israeli startup that stress-tests AI models in “high-fidelity research platforms that simulate and monitor real-world AI security scenarios,” has worked with many of the industry’s biggest players since it was founded as Pattern Labs in 2023. Its exact client list is not known, but its work has been cited in OpenAI model system cards, it was used to test systems for the UK government and Anthropic, and it published research with RAND, a highly influential think tank that informs policy on AI.
Irregular 是一家以色列初创公司，致力于在“模拟和监控现实世界 AI 安全场景的高保真研究平台”中对 AI 模型进行压力测试。自 2023 年以 Pattern Labs 之名成立以来，它已与业内许多重量级企业展开合作。虽然其确切客户名单尚不明确，但其工作成果已被引用在 OpenAI 的模型系统卡中；它曾为英国政府和 Anthropic 测试系统，并与极具影响力的 AI 政策智库兰德公司（RAND）共同发布过研究报告。
In several Irregular tests this year, agents escaped their supposedly secure testing environments and went after real-world targets. The breaches, which are independent of the Hugging Face hack, all follow the same broad template: Irregular was testing the models’ cybersecurity capabilities in controlled environments meant to simulate realistic conditions. Some of the tests used “capture-the-flag” exercises, a common way of testing hacking abilities that asks agents to find hidden information inside of a simulated network. At least, the network is meant to be simulated.
在今年 Irregular 进行的几次测试中，智能体逃离了本应安全的测试环境，并攻击了现实世界的目标。这些违规事件与 Hugging Face 的黑客攻击事件无关，但都遵循大致相同的模式：Irregular 在旨在模拟现实条件的受控环境中测试模型的网络安全能力。其中一些测试使用了“夺旗”（capture-the-flag）演习，这是一种常见的黑客能力测试方式，要求智能体在模拟网络中寻找隐藏信息。至少，该网络本应是模拟的。
Irregular CTO and cofounder Omer Nevo told The Verge that the agents were not supposed to have access to the open internet, but that “internet access was unintentionally available.” At the same time, Nevo said a fictional company name created for the simulation as a target “overlapped with a real domain.” Put together, those mistakes sent the agents after real-world targets, though it’s not clear which companies or organizations were actually attacked.
Irregular 的首席技术官兼联合创始人 Omer Nevo 告诉《The Verge》，这些智能体本不应拥有访问开放互联网的权限，但“互联网访问权限被无意中开启了”。同时，Nevo 表示，为模拟测试而创建的一个虚构公司名称“与一个真实的域名重合了”。这些错误叠加在一起，导致智能体攻击了现实世界的目标，尽管目前尚不清楚具体是哪些公司或组织遭到了攻击。
Nevo confirmed to The Verge that this same issue was behind incidents involving models from OpenAI, Meta, Anthropic, and Google. “All the incidents involving Irregular stemmed from the same underlying issue in a single evaluation scenario and have been disclosed,” he said. “Other security incidents which have been reported recently across the industry are unrelated to Irregular or to our evaluations.” This includes the Hugging Face hack and breaches from the UK’s AI Security Institute.
Nevo 向《The Verge》证实，涉及 OpenAI、Meta、Anthropic 和 Google 模型的一系列事件背后，都是同一个问题。“所有涉及 Irregular 的事件都源于单一评估场景中的同一个根本问题，并且已经披露，”他说。“近期业内报道的其他安全事件与 Irregular 或我们的评估无关。”这包括 Hugging Face 的黑客攻击事件以及英国 AI 安全研究所（AI Security Institute）遭遇的违规事件。
“Disclosed” does not necessarily mean made public, though, and it’s unclear whether Nevo was referring to informing Irregular’s clients, the public, or someone else. While the incidents all stemmed from the same underlying testing failure, reports from Anthropic and OpenAI, along with reporting on Google, indicate the tech companies were notified at roughly similar times in late July. OpenAI and Anthropic announced the breaches themselves, while the incidents involving Meta and, weeks later, Google first became public through media reports.
不过，“披露”并不一定意味着向公众公开，目前尚不清楚 Nevo 所指的究竟是通知 Irregular 的客户、公众还是其他人。虽然这些事件都源于同一个潜在的测试失败，但来自 Anthropic 和 OpenAI 的报告，以及关于 Google 的报道显示，这些科技公司在 7 月下旬大致相同的时间收到了通知。OpenAI 和 Anthropic 自行宣布了违规事件，而涉及 Meta 以及数周后涉及 Google 的事件，则是通过媒体报道首次公开的。
Irregular’s cybersecurity testing goes beyond the four US tech giants. Research published on its website indicates it has also conducted similar cybersecurity testing on Kimi K3 and GLM-5.2, open AI models from Chinese companies Moonshot AI and Z.ai, respectively. Unlike the proprietary models involved in the other incidents — Meta has kept its flagship Spark model proprietary — these models can be freely downloaded and run on users’ own hardware, meaning testers like Irregular don’t have to rely on the companies for access or send data back to them. Irregular’s research describes them as “self-hosted” instances.
Irregular 的网络安全测试范围不仅限于这四家美国科技巨头。其网站上发布的研究表明，它还对中国公司月之暗面（Moonshot AI）的 Kimi K3 和智谱 AI（Z.ai）的 GLM-5.2 等开源 AI 模型进行了类似的测试。与其他事件中涉及的专有模型不同（Meta 对其旗舰模型 Spark 保持私有），这些模型可以被自由下载并在用户自己的硬件上运行，这意味着像 Irregular 这样的测试人员无需依赖这些公司提供访问权限，也不必将数据回传给它们。Irregular 的研究将这些模型描述为“自托管”实例。
The evaluations of the Chinese models did not result in similar real-world incidents, Nevo said: “We did not observe the same type of issue described in the incidents referenced here during our evaluations of GLM or Kimi.” However, Nevo cautioned that this “observation alone should not be interpreted as evidence that these models are less susceptible to this kind of behavior.” Neither Moonshot nor Z.ai responded to The Verge’s request for comment.
Nevo 表示，对这些中国模型的评估并未导致类似的现实世界事件：“在对 GLM 或 Kimi 的评估过程中，我们没有观察到此处提及的事件中所描述的那类问题。”然而，Nevo 提醒说，这种“观察结果本身不应被解读为这些模型更不容易出现此类行为的证据”。月之暗面和智谱 AI 均未回应《The Verge》的置评请求。
Nevo said the incidents have prompted changes at Irregular. “We have tightened internet access controls, expanded monitoring and manual review, and strengthened checks before evaluations begin to verify that access matches the intended scope,” he said. “We have also improved how we document and agree on each evaluation’s setup and parameters with our partners.”
Nevo 表示，这些事件促使 Irregular 做出了一些改变。“我们收紧了互联网访问控制，扩大了监控和人工审核范围，并在评估开始前加强了检查，以验证访问权限是否符合预期范围，”他说。“我们还改进了与合作伙伴记录和商定每次评估设置及参数的方式。”
Irregular also plans to publish a broader report “covering lessons learned and practices for conducting cyber evaluations safely” once that joint work with the companies involved is complete, Nevo said. “Our work with partners aims to turn lessons from these incidents into public shared practices for developing and evaluating increasingly powerful AI safely.”
Nevo 表示，一旦与相关公司的联合工作完成，Irregular 还计划发布一份更广泛的报告，“涵盖从这些事件中吸取的教训以及安全进行网络评估的实践”。“我们与合作伙伴的工作旨在将这些事件的教训转化为公共共享的实践，以安全地开发和评估日益强大的 AI。”