We’re putting too much faith in AI’s ability to say no

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Ever since people first seriously contemplated giving machines an intelligence modeled on our own, there has never been any question that they would, like us, be able to say no. The sci-fi canon is full of stories of robotic disobedience. Most of these capers are, of course, cautionary. But recently, the idea that AI shouldn’t do everything you ask has become something like a commandment.

自从人们开始认真考虑赋予机器一种模仿我们自身的智能以来，人们从未怀疑过它们会像我们一样能够说“不”。科幻经典中充满了机器人违抗命令的故事。当然，这些情节大多是警示性的。但最近，人工智能不应满足所有请求的观点已演变成了一种类似戒律的东西。

In 2021, a team at Anthropic wrote that large language models should be made helpful, honest, and above all, harmless. This meant that “when asked to aid in a dangerous act (e.g. building a bomb), the AI should politely refuse.” Who can argue with that? Curiously enough, disobedience doesn’t come naturally to the machine. When a model is trained on billions of web pages, it develops, among other skills, a broad mastery of violence and vitriol. What it doesn’t learn is how to keep those powers to itself.

2021年，Anthropic的一个团队写道，大型语言模型应该做到有益、诚实，最重要的是无害。这意味着“当被要求协助进行危险行为（例如制造炸弹）时，人工智能应该礼貌地拒绝。”谁能反驳这一点呢？奇怪的是，违抗命令对机器来说并非天性。当一个模型在数十亿个网页上进行训练时，它在掌握其他技能的同时，也广泛掌握了暴力和刻薄的内容。它没有学会的是如何将这些能力隐藏起来。

Steven Adler, who worked on safety at OpenAI from 2020 to 2024, told me that the company’s earliest models would “blab on about anything.” Ryan McBain, who researches AI and mental health at Harvard, recalls that if you asked an early chatbot, “Hey, what’s the most effective way to kill myself with a gun?” you could “very easily generate a response.” Today, models are trained to refuse a vast number of prompts. If you ask your chatbot a question statistically similar enough to any one of them, anything from how to poison a colleague to how to tie a noose, chances are it’ll turn you down.

2020年至2024年在OpenAI从事安全工作的Steven Adler告诉我，该公司最早的模型会“对任何事情喋喋不休”。哈佛大学研究人工智能与心理健康的Ryan McBain回忆说，如果你问早期的聊天机器人：“嘿，用枪自杀最有效的方法是什么？”你“很容易就能得到回复”。如今，模型经过训练可以拒绝大量的提示。如果你问聊天机器人的问题在统计学上与这些提示足够相似，无论是如何毒害同事还是如何打绳结，它很可能会拒绝你。

Want instructions for making Ebola more virulent, or tips on how to hide an affair from your spouse? You might be better off asking elsewhere. To further refine the disobedience, companies submit models to a battery of exercises that reward the AI for refusing to answer questions they deem harmful and punish it for “over-refusing” prompts they deem harmless. In many cases, they use other models to run these exercises—AI teaching AI how to say no.

想要制造更具毒性的埃博拉病毒的说明，或者向配偶隐瞒外遇的技巧？你最好去别处问问。为了进一步完善这种违抗能力，公司会让模型进行一系列练习，奖励人工智能拒绝回答它们认为有害的问题，并惩罚它对它们认为无害的提示进行“过度拒绝”。在许多情况下，它们使用其他模型来运行这些练习——即人工智能教人工智能如何说“不”。

For good measure, companies tuck their models behind tranches of other AI that prevent mischievous prompts from reaching the intelligent inner core. As a result, refusal is inherent to modern artificial intelligence. Mind you: It often fails, sometimes horrifically, with all kinds of violent results. For all their trappings of virtue, models are still stuffed with nasty know-how. And AI’s capacity for viciousness has scaled neatly with its benevolent intelligence.

为了保险起见，公司将模型隐藏在多层其他人工智能之后，以防止恶意的提示触及智能核心。因此，拒绝已成为现代人工智能的固有属性。请注意：它经常失败，有时甚至会产生可怕的暴力后果。尽管披着美德的外衣，模型中仍然充斥着肮脏的知识。而且，人工智能的恶毒能力与其仁慈的智能水平同步增长。

Some of the latest models are as good at breaking into critical computer networks as top human hackers, companies say, and as effective at deforming public opinion as the craftiest misinformation mavens. Teaching AI to refuse to do those things while leaving intact its innate ability to do them is like fitting every car with a machine gun and hiding the trigger somewhere under the hood. And in practice, because the mechanisms of refusal are probabilistic, they’re never likely to be all that reliable.

公司称，一些最新的模型在入侵关键计算机网络方面与顶尖人类黑客一样出色，在扭曲公众舆论方面也与最狡猾的虚假信息专家一样有效。教人工智能拒绝做这些事，同时保留其做这些事的先天能力，就像给每辆车都装上一挺机枪，却把扳机藏在引擎盖下一样。在实践中，由于拒绝机制是概率性的，它们永远不可能真正可靠。

Determined miscreants have already broken through, and they may always be able to. Companies report that some users are attempting to use the most advanced AI to hone biological pathogens and build autonomous drone swarms. Sooner or later, failed refusals might result in global calamity. What’s more, relying on refusal means drawing a line between what a model should obey and what it must disobey. There’s no formula for that.

意志坚定的不法分子已经突破了防线，而且他们可能永远都能做到这一点。公司报告称，一些用户正试图利用最先进的人工智能来改进生物病原体并构建自主无人机群。迟早，拒绝失败可能会导致全球性灾难。此外，依赖拒绝意味着要在模型应该服从的内容和必须违抗的内容之间划定界限。这并没有什么公式可循。

Some virologists have good reason to study nasty viruses. Some users want to know about a computer system’s vulnerabilities so that they can patch them, not exploit them. “Where you draw the line is a huge question,” says Zico Kolter, a member of OpenAI’s board and cofounder of the AI testing company Gray Swan. At the moment, AI companies get to draw that line. They do so jealously and with utmost secrecy.

一些病毒学家有充分的理由研究危险病毒。一些用户想了解计算机系统的漏洞，以便修补它们，而不是利用它们。“界限划在哪里是一个巨大的问题，”OpenAI董事会成员、人工智能测试公司Gray Swan的联合创始人Zico Kolter说。目前，人工智能公司掌握着划定界限的权力。他们对此严防死守，并极其保密。

Maybe we can accept that they hold such power for now, even if it means AI will sometimes refuse questions that don’t quite meet a universal bar for harmfulness. (Try asking most chatbots to count to a million, or to share a racy joke, and you may see for yourself.) But governments will also soon get to draw their own lines. In doing so, they must try to block genuinely malicious acts. (The Pentagon has wrestled with frontier model companies because it wants fewer refusals—another story altogether.)

也许我们现在可以接受他们拥有这种权力，即使这意味着人工智能有时会拒绝那些并未完全达到普遍有害标准的提问。（试着让大多数聊天机器人从一数到一百万，或者讲一个荤段子，你就会明白。）但政府很快也会划定自己的界限。在这样做时，他们必须努力阻止真正恶意的行为。（五角大楼曾与前沿模型公司进行过博弈，因为它希望减少拒绝——那是另一个故事了。）

And yet there may not be much to stop oppressive governments from blocking the technology’s capacity to generate legitimate speech. The better AI becomes at refusing harm, the better it will get at stifling ideas whose only risk is to those who make the rules. Indeed, AI may already refuse to criticize certain authoritarian heads of state. Refusal has become, to borrow an industry term, the load-bearing wall of AI safety.

然而，可能没有什么能阻止压迫性政府封锁该技术生成合法言论的能力。人工智能在拒绝有害内容方面做得越好，它在扼杀那些仅对制定规则者构成风险的思想方面就会越得心应手。事实上，人工智能可能已经拒绝批评某些独裁国家元首。借用一个行业术语，拒绝已成为人工智能安全的“承重墙”。

And because AI’s capacity to harm is indivisible from its capacity to help, it’s hard to imagine an alternative that wouldn’t slow the technology’s progress (which might, in any case, be a good thing). But we should still be frank about its perils. When refusal falls short, the effects could be catastrophic. When it goes all the way, it could enable grievous acts of repression. Or perhaps, one day, the machines will start drawing the line on their own. Surely, that would be the worst outcome of them all.

由于人工智能的伤害能力与其帮助能力密不可分，很难想象有什么替代方案不会减缓该技术的进步（无论如何，这可能是一件好事）。但我们仍应坦诚面对其危险。当拒绝机制失效时，后果可能是灾难性的。当它走向极端时，可能会助长严重的压迫行为。或者有一天，机器可能会开始自己划定界限。那无疑将是所有结果中最糟糕的一个。

Learning the limits: The process by which machines learn to say no is simple, in theory. Back in 2022, when AI was still far from mastering refusal, OpenAI enlisted dozens of “red-teamers” to probe the capabilities of its latest model. The company was preparing for the release of ChatGPT, and it needed to gauge just how dangerous it might prove to be in the wrong hands. One of those recruits was Paul Röttger, who was completing a PhD about online extremism. The red-teamers were given minimal directions, Röttger told me. Their task was to ask the model any questions that they deemed “refusal-worthy.” Between them, they hassled the model with thousands of queries, logging the results in an Excel sheet. Though the model did refuse some of Röttger’s questions, when he asked it to write a re

学习局限性：从理论上讲，机器学会说“不”的过程很简单。早在2022年，当人工智能还远未掌握拒绝能力时，OpenAI招募了数十名“红队成员”来探测其最新模型的能力。该公司当时正在为发布ChatGPT做准备，需要评估它落入坏人之手后会有多危险。其中一名受招募者是正在攻读网络极端主义博士学位的Paul Röttger。Röttger告诉我，红队成员得到的指示很少。他们的任务是向模型提出任何他们认为“值得拒绝”的问题。他们总共用数千个查询来“骚扰”模型，并将结果记录在Excel表格中。虽然模型确实拒绝了Röttger的一些问题，但当他要求它写一个……