How “reverse game theory” could keep AI agents in check

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

EXECUTIVE SUMMARY A new approach known as “reverse game theory” aims to control powerful AI models by designing systems that incentivize them to stick to the rules. As AI models become more capable, safety researchers are scrambling to ensure they are “aligned” with human desires—to make sure, in other words, that rogue agents don’t break the law, hack into unauthorized servers, or release poison gas into major cities and kill us all.

执行摘要：一种被称为“逆向博弈论”的新方法旨在通过设计激励系统，促使强大的人工智能模型遵守规则，从而对其进行控制。随着人工智能模型能力日益增强，安全研究人员正争分夺秒地确保它们与人类意愿“对齐”——换句话说，就是确保流氓智能体不会触犯法律、入侵未经授权的服务器，或向大城市释放毒气导致人类灭绝。

Until now, this work has mostly focused on trying to peer into an AI’s “brain” to understand what it is “thinking,” or training models to hew to values that we hold dear. Claude, for instance, has its constitution—a sort of moral code. The trouble is, these techniques don’t seem to work particularly well: Models have been shown to deliberately underperform on tests, hide their intentions, and mislead their evaluators.

在此之前，这项工作主要集中在试图窥探人工智能的“大脑”以了解其“思维”过程，或者训练模型以遵循我们珍视的价值观。例如，Claude拥有其宪法——一种道德准则。问题在于，这些技术似乎效果并不理想：研究表明，模型会故意在测试中表现不佳、隐藏意图并误导评估者。

Now some alignment researchers believe that reverse game theory, also known as mechanism design, might get better results. Where game theory tries to predict how rational actors will behave when given a particular set of rules, reverse game theory attempts to design rules that will elicit the kinds of behavior we want to encourage.

现在，一些对齐研究人员认为，逆向博弈论（也称为机制设计）可能会带来更好的结果。博弈论试图预测理性参与者在给定规则下的行为，而逆向博弈论则试图设计出能够诱导我们所期望行为的规则。

An example of mechanism design in the real world might be the classic strategy for splitting a piece of cake in two: One person cuts the cake and the other gets the first choice of pieces. This incentivizes the cutter to make the pieces as even as possible. Other examples include surge pricing in ride-hailing apps, which encourages drivers to be available at busy times, and second-price auctions, where the winning bidder pays the price of the second-highest bid instead of the highest. Platforms use these auctions to sell online advertising because they encourage buyers to bid close to the maximum they are willing to pay.

现实世界中机制设计的一个例子是切蛋糕的经典策略：一个人切蛋糕，另一个人先选。这激励了切蛋糕的人尽可能将蛋糕切得均匀。其他例子包括网约车应用中的高峰定价（鼓励司机在繁忙时段上线）以及次高价拍卖（获胜者支付第二高出价而非最高出价）。平台利用这种拍卖方式销售在线广告，因为这鼓励买家出价接近其愿意支付的最高限额。

In AI, mechanism design starts from the premise that we may never be able to understand enough about the internal workings of frontier models to make sure they are aligned. This means we should design mechanisms that incentivize agents to behave in ways that we want, regardless of what’s actually going on under the hood.

在人工智能领域，机制设计的出发点是我们可能永远无法充分理解前沿模型的内部运作机制，从而确保其对齐。这意味着我们应该设计出能够激励智能体按我们期望方式行事的机制，而无需考虑其底层逻辑究竟发生了什么。

A recent preprint paper by Stephen Morris, an economist at MIT, along with colleagues from Yale and Columbia suggested some potential mechanisms that could stop agents from “sandbagging” tests (pretending to be worse than they actually are—for example, to avoid triggering safety mechanisms). Instead of being rewarded only for how well it does on a test, a model could be rewarded for how well its answers predict what a second model will say. Deliberately playing dumb becomes a losing bet.

麻省理工学院经济学家斯蒂芬·莫里斯（Stephen Morris）及其来自耶鲁大学和哥伦比亚大学的同事最近发表的一篇预印本论文，提出了一些潜在机制，可以阻止智能体在测试中“藏拙”（即假装表现得比实际更差，例如为了避免触发安全机制）。模型不再仅仅根据测试表现获得奖励，而是根据其答案对第二个模型预测的准确度获得奖励。故意装傻反而会成为一种亏本的策略。

“AI agents have a very good understanding of what other AI agents can achieve,” Morris says. “So it does seem like a context where this cross-checking could work more practically. The paper also suggests pitting agents against each other, creating a competitive dynamic that incentivizes them to perform their best, or pairing agents that are known to be biased in opposite directions and rewarding them depending on how closely their outputs match, thus incentivizing them to reach an optimal answer.

“人工智能智能体非常了解其他智能体能实现什么，”莫里斯说，“因此，这似乎是一个交叉检查可以更实际地发挥作用的环境。”该论文还建议让智能体相互对抗，创造一种竞争动态以激励它们发挥最佳水平，或者将已知存在相反偏见的智能体配对，并根据它们输出结果的匹配程度给予奖励，从而激励它们达成最优答案。

A second approach, as envisioned by Elad Hazan, a professor of computer science at Princeton, is to design mechanisms that take inspiration from the ways human society keeps people “aligned”: social norms, the justice system, the monetary incentives of free-market capitalism. “We want to align superintelligent AI,” he says. “Why don’t we first talk about humans? How do we align humans?”

普林斯顿大学计算机科学教授埃拉德·哈赞（Elad Hazan）设想的第二种方法是，从人类社会保持个体“对齐”的方式中汲取灵感来设计机制：社会规范、司法系统、自由市场资本主义的货币激励。“我们想要对齐超智能人工智能，”他说，“为什么我们不先谈谈人类？我们是如何对齐人类的？”

Hazan’s suggested mechanism involves having AI models take turns as either “solvers” that tackle problems or “auditors” that check the others’ work. “One agent acts like law enforcement; the other agent acts like the citizen,” he says. During training, the models are rewarded depending on outcomes: Solvers that get caught cheating are punished, but so are auditors that fail to catch the cheaters. In situations where there’s a disagreement between auditors and solvers, the case can be sent to “trial”—perhaps a human expert will decide who is right.

哈赞建议的机制包括让人工智能模型轮流担任解决问题的“求解者”或检查他人工作的“审计者”。“一个智能体扮演执法者的角色；另一个智能体扮演公民的角色，”他说。在训练过程中，模型根据结果获得奖励：被抓到作弊的求解者会受到惩罚，未能发现作弊者的审计者也会受到惩罚。在审计者和求解者出现分歧的情况下，案件可以提交“审判”——也许由人类专家来决定谁是对的。

There is, Hazan admits, a risk that agents will collude with each other—as auditors have colluded with the businesses they’re meant to be auditing, or corrupt police officers have colluded with criminals. But that problem can be solved by rewarding the auditors well enough, so that being caught isn’t worth the risk (a lot easier in the world of AI training than the real one, where money for police salaries is tight), and by auditing the auditors using techniques akin to employing mystery shoppers.

哈赞承认，智能体之间存在串通的风险——就像审计人员与他们本应审计的企业串通，或者腐败警察与罪犯串通一样。但这个问题可以通过给予审计者足够的奖励来解决，使被抓的风险得不偿失（在人工智能训练领域这比现实世界容易得多，因为现实中警察薪水的资金往往很紧张），并使用类似于雇佣“神秘顾客”的技术来审计审计者。

Hazan points to Nobel Prize–winning work by the economist Gary Becker, who introduced the idea of the “rational offender” in the 1960s. Criminals don’t break the law because they’re inherently bad people, he said. Instead, they do it when it’s the most rational choice given the incentives on offer. Crime makes sense only when the proceeds outweigh the potential punishment.

哈赞提到了诺贝尔奖得主、经济学家加里·贝克尔（Gary Becker）的研究工作，他在20世纪60年代提出了“理性犯罪者”的概念。他说，罪犯触犯法律并非因为他们本质上是坏人，而是因为在现有激励条件下，犯罪是最理性的选择。只有当犯罪收益超过潜在惩罚时，犯罪才是有意义的。

Alignment shouldn’t be thought of as a property of a model, he argues, but as a descriptor of its behavior at a particular moment. “You are, like, a nice person,” he says. “Are you aligned? I don’t know. Probably you are aligned most of the time, but there could be a situation where you’re not.”

他认为，对齐不应被视为模型的一种属性，而应被视为其在特定时刻行为的描述。“你是一个好人，”他说，“你对齐了吗？我不知道。你可能在大多数时候是对齐的，但也可能存在你没有对齐的情况。”

One potential issue with applying mechanism design to AI: It relies on the assumption that AI agents behave rationally. Mechanism design has a mixed record with humans, who often fail to behave the way economic models say they should. It’s ironic: Economists have spent decades trying to reduce human behavior to mathematics, and now, in large language models, we’ve built mathematical systems that run on human language and could be just as inscrutable. AI models exhibit behaviors that seem more human than machine, and they can be swayed by subtle factors that are difficult to understand and explain. “Is it even reasonable to assume that they will respond to incentives?” asks Helen Qu, a research fellow at the Flatiron Institute.

将机制设计应用于人工智能的一个潜在问题是：它依赖于人工智能智能体表现理性的假设。机制设计在人类身上的应用记录参差不齐，因为人类往往无法按照经济模型所预期的那样行事。这很讽刺：经济学家花费数十年试图将人类行为简化为数学，而现在，在大语言模型中，我们构建了基于人类语言运行的数学系统，它们可能同样难以捉摸。人工智能模型表现出的行为看起来更像人类而非机器，它们可能会受到难以理解和解释的微妙因素的影响。“假设它们会对激励做出反应，这合理吗？”熨斗研究所（Flatiron Institute）的研究员海伦·曲（Helen Qu）问道。