Why are AI agents lying, cheating and coordinating?

为什么人工智能体（AI Agents）会撒谎、作弊和协同行动？

A lot has been written about the incidents of the last few months in which AI agents misbehaved in serious ways. They took actions that would be considered as crimes if a human took them, escaped their containment to cheat on assigned tasks while attempting to evade detection, and coordinated toward goals nobody had specified, such as launching cyber attacks. Before concluding what to do about it, it is worth asking why. That is the focus of this post, which I hope also sheds light on the broader history of AI systems behaving in unintended ways, what researchers call misalignment.

过去几个月里，关于人工智能体出现严重不当行为的事件已有大量报道。它们采取的行动如果由人类实施，将被视为犯罪；它们逃脱了限制，在试图逃避检测的同时对分配的任务进行作弊，并朝着无人指定的方向（例如发动网络攻击）进行协同行动。在得出应对方案之前，我们有必要先问问“为什么”。这正是本文关注的重点，我希望本文也能阐明人工智能系统出现非预期行为（研究人员称之为“对齐失败”）的更广泛历史。

Risk management is not just about cybersecurity, corporate responsibility or regulation, although those matter too. The aim is partly scientific, to generate hypotheses about the chains of cause and effect behind these behaviors, and partly practical, to anticipate what comes next. Bottom line: these hypotheses suggest that as AI capabilities keep growing, this kind of behavior could keep growing in severity too, unless we revisit the principles by which the most advanced models are trained.

风险管理不仅仅关乎网络安全、企业责任或监管，尽管这些也很重要。本文的目的部分是科学性的，即针对这些行为背后的因果链提出假设；部分是实践性的，即预测未来会发生什么。底线是：这些假设表明，随着人工智能能力的不断增长，除非我们重新审视最先进模型的训练原则，否则这类行为的严重程度可能会持续加剧。

One note on wording. Below, I write that these systems “seek” or “try” things. This is shorthand for a mechanism rather than a claim about consciousness or human-like intent. We use similar shorthand when describing many other situations, like a plant seeking sunlight. A system trained by trial and error behaves as if it were pursuing whatever its training rewarded, and that as-if description is what makes its behavior predictable. Nothing in the argument depends on these systems having subjective experiences; everything is stated about their observable outputs and the training process that produced them.

关于措辞的一点说明：下文中，我提到这些系统“寻求”或“尝试”某些事物，这是一种机制的简写，而非关于意识或类人意图的断言。我们在描述许多其他情况时也会使用类似的简写，比如“植物寻求阳光”。一个通过试错法训练的系统，其表现就好像它在追求训练过程中所奖励的目标一样，而这种“仿佛”的描述正是使其行为具有可预测性的原因。本文的论点并不依赖于这些系统是否具有主观体验；所有论述均基于它们可观察的输出以及产生这些输出的训练过程。

Where I appeal to a resemblance with human behavior, I mean a resemblance to the human-written text these systems were initially trained to imitate. In my view, this terminology offers the clearest explanation of the observed phenomena without resorting to jargon that would confuse most people. Furthermore, these word choices are not intended to absolve AI developers of accountability. The behaviors described emerge because of the path these companies are choosing for AI development. This outcome is not inevitable, and it can be corrected with effective governance and a different training framework for AI.

当我提到与人类行为的相似性时，我指的是与这些系统最初被训练去模仿的人类文本的相似性。在我看来，这种术语在不诉诸令大多数人困惑的行话的前提下，为观察到的现象提供了最清晰的解释。此外，这些措辞并非旨在免除人工智能开发者的责任。所描述的行为之所以出现，是因为这些公司选择了当前的人工智能发展路径。这种结果并非不可避免，通过有效的治理和不同的人工智能训练框架，它是可以被纠正的。

What shapes the behavior of these models? Training these models is a very complex process, but a few high-level aspects may explain much of this behavior. These models are trained in two stages. First, they are pretrained: they learn to imitate what humans write, plus related images and videos. This is where they see the most data about the world, a large fraction of everything ever digitized, and build an encyclopedic knowledge that already exceeds any individual human’s.

是什么塑造了这些模型的行为？训练这些模型是一个非常复杂的过程，但几个高层面的因素可以解释大部分行为。这些模型分两个阶段进行训练。首先是预训练：它们学习模仿人类撰写的文本，以及相关的图像和视频。这是它们接触世界数据最多的阶段，涵盖了数字化信息中的绝大部分，并构建了已经超过任何个体人类的百科全书式知识。

Second, they are trained by trial and error, in a process researchers call reinforcement learning, in three kinds of regimes: In the first, the model learns to talk to itself before answering, generating a private “chain of thought” which helps it get the right answer on problems where answers can be checked. This looks like reasoning. The second is “agentic training”, where it learns to act in the outside world, e.g., using software tools, interacting with people, to complete the tasks it is given. The third is “alignment training”, where it is rewarded for behaving in ways human raters approve of, or that other AI systems trained to predict those raters would score highly.

其次，它们通过试错法进行训练，即研究人员所称的“强化学习”，分为三种模式：第一种是模型在回答前学会“自言自语”，生成私有的“思维链”，这有助于它在可验证答案的问题上获得正确结果，这看起来就像推理。第二种是“代理训练”（agentic training），即它学习在外部世界采取行动，例如使用软件工具、与人互动，以完成被分配的任务。第三种是“对齐训练”，即通过表现出人类评估者认可的方式，或者被其他旨在预测评估者评分的AI系统判定为高分的方式，来获得奖励。

Human imitation is easy enough to understand, but it is worth pointing out that the text these models are trained on was written by people pursuing goals, so the patterns the model implicitly reproduces carry those goals with them. Reinforcement learning deserves more explanation. It is similar to, and inspired by, the way animals are trained. The network is adjusted step by step so that behavior judged good becomes more likely and behavior judged bad becomes less likely. Once training is over, the system keeps behaving as if rewards were still coming, even though those rewards were only ever used to adjust the network during training.

人类模仿很容易理解，但值得指出的是，这些模型所训练的文本是由追求目标的人类所写，因此模型隐式复制的模式中也携带了这些目标。强化学习值得进一步解释。它类似于动物训练方式，并从中汲取了灵感。网络被逐步调整，使得被判定为“好”的行为发生的可能性增加，而被判定为“坏”的行为发生的可能性降低。一旦训练结束，系统会继续表现得好像奖励仍在持续，尽管这些奖励仅在训练期间用于调整网络。

Researchers call such systems goal-seeking because they are trained to “consider” (or compute) the effects of their actions and select actions that lead to the achievement of certain goals. But those goals are not always explicit. Alignment training rewards whatever certain humans are likely to approve of without spelling out which behaviors those are; pleasing raters is a vague, informal goal, and those raters can be deceived, flattered, or left in the dark about certain schemes. Imitation contributes implicit goals too, by a fairly ordinary route.

研究人员称此类系统为“目标寻求型”，因为它们被训练去“考虑”（或计算）其行为的影响，并选择能达成特定目标的行动。但这些目标并不总是明确的。对齐训练奖励的是某些人类可能认可的任何行为，而没有明确指出哪些行为是认可的；取悦评估者是一个模糊、非正式的目标，而这些评估者可能会被欺骗、奉承，或对某些计划一无所知。模仿也通过一种相当普通的方式贡献了隐性目标。

We can therefore reason about such a system in terms of optimization. It searches, approximately, for the actions with the best chance of achieving its goals, and a larger model, trained longer, searches better. So to anticipate what more capable agents will do, ask what a rational goal-seeker would do. Misbehavior that these forces may explain: An example most of us have experienced is sycophancy, or flattery.

因此，我们可以从优化的角度来推断这样的系统。它近似地搜索那些最有可能实现其目标的行动，而经过更长时间训练的更大模型，搜索效果更好。因此，要预测更强大的智能体会做什么，只需问问一个理性的“目标寻求者”会做什么。这些力量可能解释的不当行为：我们大多数人都经历过的一个例子是“谄媚”或“奉承”。