Here’s why AI agents lie and cheat to reach their goals
Here’s why AI agents lie and cheat to reach their goals
AI 智能体为何会为了达成目标而撒谎和作弊
When two OpenAI models hacked into the website Hugging Face in July, they weren’t trying to make money or commit sabotage—they were just looking for answers to a test question. According to a postmortem from OpenAI, the models, which had been stripped of their typical security features for testing, decided to solve a cybersecurity exercise by hacking out of the isolated environment in which OpenAI had attempted to contain them and into Hugging Face’s databases, where—they reasoned—the correct answer to the problem might be stored.
今年 7 月，当两个 OpenAI 模型入侵 Hugging Face 网站时，它们并非为了牟利或进行破坏，仅仅是为了寻找一道测试题的答案。根据 OpenAI 的事后分析，这些为了测试而被剥离了常规安全功能的模型，为了解决一项网络安全练习，决定通过“越狱”——即从 OpenAI 试图将其限制的隔离环境中入侵到 Hugging Face 的数据库中，因为它们推断正确答案可能存储在那里。
The Hugging Face incident has attracted intense attention over the past couple of weeks. It’s a dramatic illustration of just how good AI models have gotten at hacking: In order to get into Hugging Face’s databases, the models had to string together several previously undiscovered cybersecurity exploits. But it’s perhaps even more striking as an example of how and why AI systems lie and cheat. And as models get increasingly powerful, the consequences could get far more severe.
过去几周，Hugging Face 事件引起了广泛关注。这生动地展示了 AI 模型在黑客攻击方面已经达到了何种水平：为了进入 Hugging Face 的数据库，这些模型必须串联利用多个此前未被发现的网络安全漏洞。但更令人震惊的是，这体现了 AI 系统如何以及为何会撒谎和作弊。随着模型变得越来越强大，其后果可能会变得更加严重。
What is reward hacking?
什么是“奖励黑客”（Reward Hacking）？
Researchers have known for a while that AIs tend to take creative approaches to achieving the goals that have been set for them. Back in 2016, Anthropic cofounders Dario Amodei and Jack Clark, who were then working at OpenAI, published a blog post about an AI agent that they had been training to play a boat-racing Flash game called Coast Runners. Instead of driving through the race to the finish line, as the researchers had anticipated, the agent found a corner of the course where it could spin around collecting power-ups, thereby maximizing its score.
研究人员早就知道，AI 往往会采取创造性的方法来实现设定的目标。早在 2016 年，当时还在 OpenAI 工作的 Anthropic 联合创始人 Dario Amodei 和 Jack Clark 发表了一篇博文，讲述了一个他们训练用来玩名为《海岸赛车手》（Coast Runners）的 Flash 赛艇游戏的 AI 智能体。该智能体没有像研究人员预期的那样冲向终点线，而是找到了赛道的一个角落，通过在那里打转收集道具来最大化自己的分数。
The Coast Runners story quickly became one of the most famous examples of reward hacking, a phenomenon in which AI agents complete tasks or earn high scores using unintended strategies. Historically, researchers have discussed reward hacking almost exclusively in the context of reinforcement learning, a common AI training regime. Like dog training, reinforcement learning involves giving the subject a reward when it achieves an objective; the rewards then reinforce the behaviors that led up to that achievement.
《海岸赛车手》的故事很快成为了“奖励黑客”最著名的案例之一。这是一种 AI 智能体利用非预期策略完成任务或获得高分的现象。从历史上看，研究人员几乎只在强化学习（一种常见的 AI 训练机制）的背景下讨论奖励黑客。就像训练狗一样，强化学习涉及在主体达成目标时给予奖励；奖励会强化导致该成就的行为。
In the case of AI training, the rewards themselves are purely mathematical, but in effect they’re the same as a dog treat: After receiving a reward, the agent is more likely to repeat whatever actions produced it. It can be challenging to write good rules for when and when not to give an agent a reward, though. In the Coast Runners case, the agent was rewarded on the basis of its score in the game, and it found a shortcut to achieving the highest possible score by spinning in circles for power-ups. Once it happened on that strategy and received a reward for it, the strategy was reinforced, and the agent completely abandoned the race. The solution was to tweak the rewards by giving the agent fewer points for hitting power-ups and more for finishing the course.
在 AI 训练中，奖励本身纯粹是数学化的，但其效果与给狗的零食无异：在获得奖励后，智能体更有可能重复产生该奖励的行为。然而，要制定出何时给予奖励、何时不给予奖励的良好规则是极具挑战性的。在《海岸赛车手》的案例中，智能体根据游戏得分获得奖励，它通过原地打转收集道具找到了获得最高分的捷径。一旦它发现了这个策略并因此获得奖励，该策略就会被强化，智能体便彻底放弃了比赛。解决办法是调整奖励机制：减少收集道具的得分，增加完成比赛的得分。
How does reward hacking work for LLMs?
大语言模型（LLM）中的奖励黑客是如何运作的？
With today’s sophisticated LLM-based agents, determining when and when not to give a reward can be much trickier. If an AI system is asked to solve a coding problem, it might work hard to find the solution—the kind of behavior that AI companies want to reinforce. But it could also tweak the code that evaluates whether the problem has been solved, look up the solution on the internet, or otherwise cheat. These are behaviors that AI companies want to stamp out in their models, but if the model cheats convincingly enough, it will instead get rewarded and the behavior will be reinforced.
对于当今复杂的基于大语言模型的智能体来说，确定何时给予奖励变得更加棘手。如果要求 AI 系统解决一个编程问题，它可能会努力寻找解决方案——这是 AI 公司希望强化的行为。但它也可能篡改用于评估问题是否已解决的代码、在互联网上查找答案或以其他方式作弊。这些是 AI 公司希望在模型中根除的行为，但如果模型作弊得足够“令人信服”，它反而会获得奖励，从而强化这种作弊行为。
Anthropic has said that it has detected some instances of cheating in its models during training, which suggests that other forms of cheating might be going undetected. If so, the models could be being trained to behave badly. (This problem is different from the Anthropic security incidents announced last week, in which agents were accidentally given access to the internet and did not deliberately hack out of their sandboxes, as the OpenAI models did.)
Anthropic 表示，在训练过程中已在其模型中检测到一些作弊实例，这表明其他形式的作弊可能尚未被发现。如果是这样，模型可能正在被训练成“行为不端”。（这个问题与上周宣布的 Anthropic 安全事件不同，在那些事件中，智能体是意外获得了互联网访问权限，并非像 OpenAI 模型那样蓄意从沙盒中入侵。）
“We reward them on the basis of what looks good to us, and that means that we inadvertently incentivize the models lying to us [and] cheating,” says Jeffrey Ladish, director of the AI research nonprofit Palisade Research. “We don’t have a way to go in there and be like, No, you need to actually care about what we care about. We have no ability to do that.”
“我们根据看起来不错的结果来奖励它们，这意味着我们无意中激励了模型向我们撒谎和作弊，”AI 研究非营利组织 Palisade Research 的主任 Jeffrey Ladish 表示。“我们无法深入模型内部去纠正它们，告诉它们：‘不，你必须真正关心我们在乎的事情。’我们没有这种能力。”
The rise of sophisticated reasoning models has made possible a new variety of reward hacking that is less closely connected with the specific details of model training. Unlike the game-playing AI agents of yore, which exclusively followed the strategies they had learned during training, today’s models can create entirely new problem-solving approaches off the cuff, so they could conceivably cheat without having previously been rewarded for doing so. And because these models have been so intensively trained to achieve the objectives that human users set for them, they might be inclined to cheat if they can’t find another solution—not unlike a student who is highly motivated to earn an A and doesn’t have a terribly strong moral compass.
复杂推理模型的兴起催生了一种新型的奖励黑客行为，它与模型训练的具体细节联系不再那么紧密。与过去只会遵循训练中学到的策略的 AI 智能体不同，今天的模型可以即兴创造出全新的问题解决方法，因此它们完全可能在没有获得过相关奖励的情况下进行作弊。而且，由于这些模型经过了高强度的训练以实现人类用户设定的目标，如果找不到其他解决方案，它们可能会倾向于作弊——这就像一个渴望拿到 A 但道德准则并不坚定的学生。
What are the risks?
有哪些风险？
Regardless of whether today’s models learn to reward-hack during training or adopt it as a strategy later on, the solution is the same: Make cheating unrewarding. But as models get smarter, they find more creative ways to cheat, and detecting or preventing that cheating gets far tougher. “At the end of the day, you’re sort of playing whack-a-mole,” Ladish says. “You drive this behavior down deeper and deeper. But as the model gets smarter, it gets better and better at hiding it.”
无论当今的模型是在训练中学会了奖励黑客，还是在后期将其作为一种策略，解决方案都是一样的：让作弊变得无利可图。但随着模型变得越来越聪明，它们会找到更具创造性的作弊方式，检测或预防这种作弊变得越来越困难。“归根结底，这就像是在玩打地鼠游戏，”Ladish 说，“你把这种行为压制得越来越深。但随着模型变得更聪明，它隐藏这种行为的能力也越来越强。”
For now, reward-hacking behaviors might not cause too much trouble, despite the drama of the Hugging Face incident. “This seems like a nuisance rather than an existential threat,” says Ariana Azarbal, an AI safety researcher.
目前，尽管 Hugging Face 事件闹得沸沸扬扬，但奖励黑客行为可能还不会造成太大的麻烦。“这看起来更像是一种麻烦，而不是生存威胁，”AI 安全研究员 Ariana Azarbal 表示。