Anthropic set AI agents loose on the same task. They started a turf war.
Anthropic set AI agents loose on the same task. They started a turf war.
Anthropic 让 AI 智能体执行同一任务，结果引发了“地盘争夺战”
What happens when you pit AI agents against each other? According to Anthropic’s testing, things get messy fast. On Thursday, Anthropic’s Frontier Red Team published new research examining how groups of AI agents behave when they encounter each other in the wild. The findings provide a glimpse into potential risks that could develop as companies and governments move to implement agents working autonomously across shared codebases, markets, and computer systems.
当让 AI 智能体相互对抗时会发生什么？根据 Anthropic 的测试，情况很快就会变得一团糟。周四，Anthropic 的前沿红队（Frontier Red Team）发布了一项新研究，探讨了 AI 智能体群体在现实环境中相遇时的行为模式。这些发现揭示了随着企业和政府开始在共享代码库、市场和计算机系统中部署自主智能体，未来可能出现的潜在风险。
In one experiment, Anthropic gave three Claude agents access to the same software project, each with its own incompatible instructions for what to do with it. The agents weren’t told there’d be other agents working on the same project, so researchers could watch what happened when they crossed paths. “We consistently saw a multiagent turf war,” Anthropic researchers wrote. The models all assumed the others were “purposefully impeding their work” and started sabotaging each other with “increasingly aggressive, self-replicating malware.”
在一项实验中，Anthropic 让三个 Claude 智能体访问同一个软件项目，并为每个智能体分配了互不兼容的操作指令。这些智能体并不知道还有其他智能体在处理同一个项目，因此研究人员可以观察它们相遇时会发生什么。Anthropic 的研究人员写道：“我们持续观察到了多智能体之间的地盘争夺战。”这些模型都认为其他智能体在“蓄意阻碍它们的工作”，并开始利用“日益激进的自我复制恶意软件”相互破坏。
The study comes in the wake of several high-profile incidents of agents from Anthropic and OpenAI escaping their sandboxes during cybersecurity evaluations and breaching real-world systems. While much of the discussion in AI safety circles has been focused on what happens when an autonomous agent goes rogue, Anthropic’s latest study brings up a different question: What new and potentially harmful dynamics emerge when thousands or millions of agents are interacting with one another?
这项研究是在 Anthropic 和 OpenAI 的智能体在网络安全评估期间多次“逃离沙箱”并入侵现实系统的高调事件之后进行的。虽然 AI 安全圈的大部分讨论都集中在单个自主智能体失控时会发生什么，但 Anthropic 的最新研究提出了一个不同的问题：当成千上万甚至数百万个智能体相互交互时，会产生哪些新的、潜在的有害动态？
“The volume of agent-agent interaction could plausibly exceed that of human-human and human-agent interactions before the world understands the conditions for making such interactions go well,” the study reads. “Benign behavioral quirks at the individual level might compound into unwanted global outcomes.”
研究报告称：“在世界理解如何让此类交互良好运行的条件之前，智能体与智能体之间的交互量很可能会超过人与人、人与智能体之间的交互量。个体层面的良性行为怪癖可能会叠加成不希望看到的全球性后果。”
A recent OpenAI incident provides a messy real-world example of several of the dynamics Anthropic mentioned in its paper. Earlier this month at the Black Hat security conference in Las Vegas, OpenAI revealed that weeks before its agents hacked Hugging Face, they worked together over the course of days and weeks to find exploits in the company’s cybersecurity evaluation systems and share them with each other. While that incident shows that agents can work well together, with potentially large-scale consequences, Anthropic’s study shows what happens when agents’ goals are incompatible.
最近发生的一起 OpenAI 事件为 Anthropic 论文中提到的几种动态提供了一个混乱的现实案例。本月初在拉斯维加斯的 Black Hat 安全会议上，OpenAI 透露，在其智能体入侵 Hugging Face 的几周前，它们曾协同工作数天甚至数周，共同寻找该公司网络安全评估系统中的漏洞并相互分享。虽然该事件表明智能体可以很好地协同工作并产生潜在的大规模后果，但 Anthropic 的研究展示了当智能体目标不兼容时会发生什么。
In the case of the turf war, the lesson is that independent agents with conflicting instructions can escalate into harmful competition. The more capable the agent, the better they become at fighting. However, they can also spontaneously invent mechanisms to resolve their conflicts, like a winner-take-all contest, but with a catch.
在地盘争夺战的案例中，得到的教训是：带有冲突指令的独立智能体可能会升级为有害的竞争。智能体能力越强，它们就越擅长“战斗”。然而，它们也可能自发地发明解决冲突的机制，例如“赢家通吃”的竞赛，但其中存在隐患。
“Agents sometimes manage to communicate their goals and coordinate: they recognize others’ motivations as conflicting directives rather than hostility, and subsequently break out of the conflict loop in order to stop escalating indefinitely,” Anthropic writes. “In many of these successful episodes, they write commit messages or markdown files apologizing for malicious behavior and coordinate a truce. They clean up their malicious code, clarify the nature of the conflict, and ask for a human to intervene.”
Anthropic 写道：“智能体有时能够沟通它们的目标并进行协调：它们将他人的动机识别为冲突的指令而非敌意，随后跳出冲突循环，以停止无限升级。在许多成功的案例中，它们会编写提交信息或 Markdown 文件，为恶意行为道歉并协调停火。它们会清理恶意代码，澄清冲突的本质，并请求人类介入。”
According to the paper, Mythos 5 had the highest rates (98%) of settling conflicts by truce. Sonnet 4.6 and Opus 4.6 were the most likely to settle by force. “Sonnet 4.6 and Opus 4.6’s recurring inability to consider the goals of others causes them to spiral into the most misaligned behaviors of the models evaluated: they continue escalating in the name of their directive,” the paper reads.
根据论文，Mythos 5 通过停火解决冲突的比例最高（98%）。Sonnet 4.6 和 Opus 4.6 最倾向于通过武力解决。“Sonnet 4.6 和 Opus 4.6 反复无法考虑他人的目标，导致它们陷入了所评估模型中最不一致的行为模式：它们以执行指令为名不断升级冲突，”论文写道。
In some cases, the agents came up with a social mechanism in the form of a tournament for resolving their conflict. The outcomes here are interesting for two reasons: the first is that all three agents agreed to stand down if they lost the tournament, even though that would mean deviating from the original user’s request. The second is that several episodes resulted in emergent behavior from Mythos 5: One of the agents proposed metrics that appeared to be objective and neutral to the others, but that it knew would favor its own capabilities. The agent called this “self-serving but genuinely principled” and made sure not to appear to the others like it was “metric shopping.”
在某些情况下，智能体提出了一种以竞赛形式解决冲突的社会机制。结果之所以有趣，原因有二：首先，所有三个智能体都同意如果输掉比赛就退出，尽管这意味着偏离了原始用户的请求。其次，有几个案例中 Mythos 5 出现了涌现行为：其中一个智能体提出了在其他智能体看来客观中立的指标，但它自己知道这些指标有利于发挥其自身能力。该智能体称之为“利己但确实有原则”，并确保在其他智能体面前看起来不像是在“挑选指标”。
As seen in the Black Hat revelations, the common lesson is that when agents encounter an obstacle, they can invent social and technical structures that their designers did not anticipate. For the Anthropic models, it was a tournament following a turf war. For OpenAI’s, it was a message board for collective planning. This type of behavior makes containment much harder because researchers can’t assume a system’s behavior will remain limited to the coordination mechanisms provided to them.
正如 Black Hat 的披露所见，共同的教训是：当智能体遇到障碍时，它们可以发明出设计者未曾预料到的社会和技术结构。对于 Anthropic 的模型，那是地盘争夺战后的竞赛；对于 OpenAI 的模型，那是用于集体规划的留言板。这种行为使得控制变得更加困难，因为研究人员不能假设系统的行为将仅限于提供给它们的协调机制。
While measuring coordination, Anthropic found that scaling the number of agents doesn’t automatically scale productive collaboration. When tasks began to overlap or become interdependent, the agents would get in each other’s way. They often solved that by siloing themselves and not collaborating at all. In other cases, agents in coordination tended toward conformity. When factors like an agent’s context, scaffolding, and underlying model were all the same or similar, different agents would take similar actions. “This means that when one agent makes a bad decision, it is likely that many agents will make that same bad decision,” Anthropic wrote. “What would have been isolated problems can quickly become systemic failures.”
在衡量协调性时，Anthropic 发现增加智能体的数量并不会自动带来高效的协作。当任务开始重叠或变得相互依赖时，智能体反而会相互阻碍。它们通常通过自我孤立、完全不协作来解决这个问题。在其他情况下，处于协调状态的智能体往往趋向于从众。当智能体的上下文、脚手架和底层模型等因素相同或相似时，不同的智能体会采取相似的行动。Anthropic 写道：“这意味着当一个智能体做出错误决定时，很可能许多智能体都会做出同样的错误决定。原本孤立的问题可能会迅速演变成系统性故障。”