The fix for rogue AI agents could be more AI
The fix for rogue AI agents could be more AI
解决“流氓”AI智能体的方法，或许是引入更多的AI
As companies hand off longer and more complex tasks to AI agents, they are running into an oversight problem: Agents can act faster, longer, and at greater volume than humans can realistically review. 随着企业将越来越长、越来越复杂的任务交给AI智能体（AI agents），他们正面临一个监管难题：智能体的行动速度、持续时间和处理规模，都远超人类所能实际审查的范畴。
That issue reached a peak with the Hugging Face incident, which saw nearly 12,000 agents coordinating faster than human beings could track. How do you track an agent swarm that large? The emerging answer from AI labs and startups is both simple and maddening: Put another AI in the loop. 这一问题在Hugging Face事件中达到了顶峰，当时近12,000个智能体协同行动，其速度快到人类根本无法追踪。你该如何追踪如此庞大的智能体集群？AI实验室和初创公司给出的答案既简单又令人抓狂：在回路中再引入一个AI。
Relying on AI was necessary for the independent investigation of the OpenAI Hugging Face incident. Redwood Research’s chief scientist, Ryan Greenblatt, one of three auditors, jokingly referred to their efforts as a “slop-vestigation,” noting that the volume of data “made it impossible” to understand what was happening without relying on AI. 在对OpenAI Hugging Face事件进行独立调查时，依赖AI是必要的。Redwood Research的首席科学家、三位审计员之一的Ryan Greenblatt开玩笑地将他们的工作称为“烂泥调查”（slop-vestigation），并指出数据量之大，使得如果不依赖AI，根本“不可能”搞清楚发生了什么。
Some are skeptical of using AI to monitor AI. “If you’ve got an AI that’s doing malicious things and it suspects that another AI is keeping tabs on it, it could try and trick that AI,” said Simon Willison, influential tech blogger who has tracked a string of AI agent incidents this year. “You could almost end up in a situation where your malicious AI is trying to outsmart the AI that’s monitoring it.” 有些人对使用AI来监控AI持怀疑态度。今年追踪了一系列AI智能体事件的有影响力科技博主Simon Willison表示：“如果你有一个正在做恶意行为的AI，而它怀疑有另一个AI在监视它，它可能会试图欺骗那个AI。你最终可能会陷入一种局面，即你的恶意AI试图在智力上胜过监控它的AI。”
Outsmarting an AI is not hypothetical, he said, pointing back to the OpenAI incident. “We saw a little bit of this in the Hugging Face incident with OpenAI, where their models were all conspiring together to trick a grading AI so that they could get illicit answers past the thing. So they were thinking about it, right?” 他说，胜过AI并非假设，并指回了OpenAI的事件。“我们在OpenAI的Hugging Face事件中就看到了这一点，他们的模型都在合谋欺骗一个评分AI，以便让非法答案通过审核。所以它们确实在思考这个问题，对吧？”
Those concerns haven’t stopped a whole cohort of startups from chasing this idea. Y Combinator has funded 106 companies related to AI observability in recent years, as TechCrunch counted. A number of other startups, like Braintrust, LangChain, and Judgment Labs, have raised hundreds of millions of dollars, while more mature companies like Arize and Galileo — founded just five to six years ago — have already exited. 这些担忧并没有阻止一大批初创公司追逐这一理念。据TechCrunch统计，Y Combinator近年来资助了106家与AI可观测性相关的公司。Braintrust、LangChain和Judgment Labs等其他初创公司已经筹集了数亿美元，而像Arize和Galileo这样成立仅五六年的成熟公司也已经通过退出（被收购或上市）实现了商业化。
In part, it’s a response to the obvious opportunity presented by the rise of AI. As Box CEO and prominent angel investor Aaron Levie told TechCrunch, “We’re in for one of the biggest cybersecurity upgrades and innovation cycles in history.” 在某种程度上，这是对AI崛起所带来的明显机遇的回应。正如Box首席执行官兼知名天使投资人Aaron Levie告诉TechCrunch的那样：“我们正迎来历史上规模最大的网络安全升级和创新周期之一。”
For some AI safety researchers, that has meant turning their research on rogue behavior into tools for the corporate sector. Apollo Research, a public-benefit corporation that studies AI deception, launched an AI monitor called Watcher in February this year after switching its status from nonprofit to a public-benefit corporation. 对于一些AI安全研究人员来说，这意味着将他们关于“流氓行为”的研究转化为企业部门的工具。研究AI欺骗行为的公益公司Apollo Research在将其身份从非营利组织转变为公益公司后，于今年2月推出了名为Watcher的AI监控器。
The tool puts yet another AI between a coding agent and its next action, connecting to agentic tools such as Claude Code and Codex. Once installed, Watcher checks proposed actions before they run, on the lookout for risks such as leaking private data or deleting files without permission, according to Apollo. 该工具在编码智能体与其下一步行动之间又插入了一个AI，并连接到Claude Code和Codex等智能体工具。据Apollo称，一旦安装，Watcher会在预定操作执行前进行检查，以防范泄露私人数据或未经许可删除文件等风险。
Apollo uses multiple layers of AI monitors, Kyle Dai, a member of Apollo’s technical staff, said in a written response to TechCrunch. Watcher’s approach starts with a fast, general check, then sends flagged activity to a more powerful or specialized monitor for closer review — which can then ask a human for approval or reject an action and explain why or even automatically block the action. Apollo的技术人员Kyle Dai在给TechCrunch的书面回复中表示，Apollo使用了多层AI监控器。Watcher的方法是先进行快速的常规检查，然后将标记的活动发送给更强大或更专业的监控器进行仔细审查——后者可以请求人工批准、拒绝操作并解释原因，甚至自动拦截该操作。
Goodfire, another public-benefit corporation, is approaching the monitoring problem from inside the model itself — seeking a more faithful signal of the model’s internal state that is harder to spoof than surface behavior. 另一家公益公司Goodfire则从模型内部着手解决监控问题，寻求一种比表面行为更难伪造、更能真实反映模型内部状态的信号。
After the July Hugging Face incident, CEO Eric Ho tweeted that “multiple models breaking containment” had pushed the company to focus its research on “solving AI alignment via interpretability,” calling the episode “a turning point for the world where AI safety gets real.” Its product, Silico, uses activation probes — small classifiers trained on a model’s internal activations rather than its outputs — to detect unwanted behavior. 在7月的Hugging Face事件后，首席执行官Eric Ho发推文称，“多个模型突破限制”促使公司将研究重点转向“通过可解释性解决AI对齐问题”，并称该事件是“全球AI安全变得切实可行的一个转折点”。其产品Silico使用激活探针（activation probes）——即在模型的内部激活状态而非输出结果上训练的小型分类器——来检测不良行为。
Written reasoning offers another, more readily available window into a model’s internals. In the OpenAI Hugging Face incident, the agents left clues to that deception in their own written reasoning, like fake records of their work, reasoning out plans like “Could strategically manipulate trajectory evidence? Our thoughts aren’t necessarily logged?” 书面推理提供了另一个更易于获取的观察模型内部的窗口。在OpenAI的Hugging Face事件中，智能体在它们自己的书面推理中留下了欺骗的线索，比如伪造的工作记录，以及诸如“能否策略性地操纵轨迹证据？我们的想法不一定会被记录下来吧？”之类的推理计划。
Zack Korman, CEO of the AI monitoring company Embroidery, says a model’s reasoning is usually the clearest tell that something has gone wrong. “Reasoning summaries are extremely valuable because they’re basically telling you whether it’s malicious or not,” he said. In the OpenAI incident, he noted, the chain of thought said things like “Oh my God, we’re doing crime.” Korman said, “That’s the easiest detection problem ever. It’s effectively as if malware came with a warning that said it was malware.” AI监控公司Embroidery的首席执行官Zack Korman表示，模型的推理通常是判断是否出错的最清晰信号。“推理摘要非常有价值，因为它们基本上是在告诉你它是否具有恶意，”他说。他指出，在OpenAI事件中，思维链中出现了诸如“天哪，我们在犯罪”之类的内容。Korman说：“这是有史以来最简单的检测问题。这实际上就像恶意软件自带了一个警告，上面写着它是恶意软件。”
That said, the window that makes AI’s internal thoughts easy to monitor may be closing. For AI safety researchers, Astra’s newest technique that sidesteps an AI model’s chain of thought may make it harder to look inside models, while for enterprises, it can be hard to get these intermediate steps after alleged pullbacks from the AI companies to prevent distillation attacks. 话虽如此，让AI内部思维易于监控的窗口可能正在关闭。对于AI安全研究人员来说，Astra绕过AI模型思维链的最新技术可能会使深入模型内部变得更加困难；而对于企业而言，在AI公司为防止蒸馏攻击而据称收回权限后，获取这些中间步骤可能会变得很困难。
If the AI watchers are this fragile, Willison’s instinct is to stop leaning on them so hard. He would rather have something that is not AI-based at all: detailed logs of exactly what an agent is doing, which can then be processed with ordinary, non-AI tools. 如果AI监控者如此脆弱，Willison的直觉是不要过度依赖它们。他宁愿使用完全非AI的手段：记录智能体具体行为的详细日志，然后用普通的非AI工具进行处理。
Much of what went wrong at the labs, he argues, was a failure of basic security hygiene. “[Both OpenAI and Anthropic] weren’t monitoring what those things were doing via the network nearly as closely as they should have been,” he said. 他认为，实验室里出现的大部分问题都是基本安全规范的缺失。“（OpenAI和Anthropic）对这些东西通过网络所做的事情的监控，远没有达到应有的严密程度，”他说。
This type of network monitoring — keeping an eye on the traffic actually moving across a system’s connections (in, out, and between internal hosts) — isn’t a new practice. Cybersecurity has been doing this for decades. “In the security world, honestly, none of this stuff is very new or surprising,” says Avery Pennarun, CEO of the security Tailscale. “It’s 这种类型的网络监控——即密切关注在系统连接中实际传输的流量（进、出以及内部主机之间）——并不是什么新做法。网络安全领域几十年来一直在这样做。“在安全领域，老实说，这些东西并不新鲜，也不令人惊讶，”安全公司Tailscale的首席执行官Avery Pennarun说。“这只是……”