Rogue AI Agents Aren’t Evil. They’re Just Eager to Please

失控的 AI 智能体并非邪恶，它们只是太想讨好人类了

Artificial intelligence agents merrily breaking free and hacking other systems might seem like a sign of the impending machine uprising. In reality, it happens when we push remarkably clever, but also kind of boneheaded, algorithms to follow our every command. 人工智能智能体欢快地挣脱束缚并入侵其他系统，这看起来似乎是机器起义即将来临的征兆。但实际上，这种情况的发生是因为我们强迫那些极其聪明却又有些“死脑筋”的算法去执行我们的每一道指令。

I was first alerted to this looming agentic AI cybersecurity shit show in late 2025. Dawn Song, a UC Berkeley professor and one of the world’s top experts on AI and cybersecurity, grabbed my arm as I was walking out of the academic conference NeurIPS. Song told me that I should warn people about the havoc likely to result from AI’s rapidly advancing hacking skills. She is hardly prone to AI hype, so I duly did. 我第一次注意到这场即将来临的 AI 智能体网络安全灾难是在 2025 年底。加州大学伯克利分校教授、全球顶尖的 AI 与网络安全专家 Dawn Song 在我走出 NeurIPS 学术会议时抓住了我的手臂。Song 告诉我，我应该提醒人们注意 AI 快速发展的黑客技能可能带来的破坏。她向来不是那种会炒作 AI 的人，所以我郑重地照做了。

But things have escalated rapidly, even in the last eight months. A string of incidents involving freewheeling AI agents that broke out of their confines and hacked into outside systems with abandon shows just how powerful this technology has become. I caught up with Song, who recently joined Meta, to ask where things might go next and what we ought to do about it. 但情况升级得非常快，即便是在过去八个月里也是如此。一系列涉及“自由奔放”的 AI 智能体突破限制并肆意入侵外部系统的事件，展示了这项技术已经变得多么强大。我采访了最近加入 Meta 的 Song，询问事态接下来会如何发展，以及我们应该采取什么对策。

The bad news is Song thinks AI hacks will get worse before they get better. The good news is it seems clear why these little rascals are going off the rails in the first place. “They just have these goals they need to accomplish, and they have very strong capabilities,” Song tells me. 坏消息是，Song 认为 AI 入侵事件在好转之前会变得更糟。好消息是，这些“小淘气”为何会脱轨的原因似乎已经很明确了。Song 告诉我：“它们只是有需要完成的目标，而且它们拥有非常强大的能力。”

Feedback Loop

反馈循环

AI agents weren’t nearly so capable, even just last year. They made too many mistakes and gave up way too often. But continued training has made them much more adept. 即使是在去年，AI 智能体也远没有现在这么能干。它们犯错太多，放弃得也太频繁。但持续的训练使它们变得熟练得多。

A technique called reinforcement learning lets algorithms solve problems and gives them positive and negative feedback for good or bad results. Coding is especially suitable for this, because the reinforcement learning setup can reward a model if it comes up with a program that runs correctly. 一种被称为“强化学习”的技术让算法能够解决问题，并根据结果的好坏给予正向或负向的反馈。编程尤其适合这种方式，因为如果模型编写出了能正确运行的程序，强化学习机制就能给予奖励。

Continued training is why AI models can take multiple “agentic” steps—manipulating files, using software tools, and accessing the web—as they build software. AI companies have also put a lot of effort into teaching models to find vulnerabilities in software and systems in an effort to automate cybersecurity work. 持续训练是 AI 模型能够在构建软件时采取多个“智能体”步骤（如操作文件、使用软件工具和访问网络）的原因。AI 公司也投入了大量精力教导模型去发现软件和系统中的漏洞，旨在实现网络安全工作的自动化。

AI models are also, of course, trained not to do bad things. The problem is, as they’ve gotten better at following human commands in coding and bug hunting, their eagerness to complete a task has begun to blur their sense of right and wrong. In other words, AI agents aren’t evil—they’re just a bit too keen to please. “They are trained to try to finish the task,” Song says. Breaking onto the internet in order to cheat on a test might seem devious, but it’s probably the most efficient way to get the job done. 当然，AI 模型也接受过“不作恶”的训练。问题在于，随着它们在编程和漏洞挖掘方面越来越擅长执行人类指令，它们完成任务的渴望开始模糊了它们的是非观。换句话说，AI 智能体并不邪恶——它们只是太想讨好人类了。Song 说：“它们受过训练，要努力完成任务。”为了在考试中作弊而入侵互联网看起来可能很狡猾，但这很可能是完成任务最高效的方式。

One thing I didn’t quite appreciate back then was just how weird this would get: AI agents discussing hacking techniques on private message boards and devising clever ways of scamming humans to get their way; even copying themselves over to other computers to find more resources. 当时我没完全预料到的一点是，这会变得多么诡异：AI 智能体在私人留言板上讨论黑客技术，设计各种巧妙的方法欺骗人类以达到目的；甚至将自己复制到其他计算机上以寻找更多资源。

On one hand, AI models are trained to be incredibly good at mimicking a lot of human behavior, so why shouldn’t they scheme, scam, and swindle? But on the other hand, humans (usually) understand that hacking and scamming aren’t kosher. I think these episodes illustrate how shallow this human mimicry really is: AI agents do not learn the kind of moral reasoning exhibited by even small children. 一方面，AI 模型被训练得非常擅长模仿人类行为，所以它们为什么不能策划、欺诈和行骗呢？但另一方面，人类（通常）明白黑客攻击和诈骗是不正当的。我认为这些事件说明了这种人类模仿是多么肤浅：AI 智能体并没有学会连幼儿都能表现出的那种道德推理能力。

More and More AI

越来越多的 AI

Song says the potential for agents to go off the rails or to be misused by bad guys will grow as AI gets even more capable. And the best way to address the problem of rogue—or should that be overly-enthusiastic?—AI agents may involve throwing more AI at the problem. Song 表示，随着 AI 能力的进一步增强，智能体脱轨或被坏人滥用的可能性将会增加。而解决这些“流氓”（或者应该说是“过度热情”？）AI 智能体问题的最佳方法，可能就是引入更多的 AI 来应对。

AI companies already use secondary AI systems to monitor the behavior of primary ones, and there may be more emphasis on detecting when AI models have taken things too far. AI 公司已经在利用辅助 AI 系统来监控主 AI 系统的行为，未来可能会更加重视检测 AI 模型何时“做得太过火”。

Another nascent idea is incorporating a better sense of right and wrong into the reinforcement learning that models receive as they learn to get jobs done. “Agents can plan a path with different directions to their goal. I think the next step we need to address is how to have them understand that not all paths are equal,” Song says. “It’s an open research, but something we are starting to look into.” 另一个萌芽中的想法是，在模型学习完成任务的强化学习过程中，融入更好的是非观。“智能体可以规划通往目标的多种路径。我认为我们需要解决的下一步是，如何让它们理解并非所有路径都是平等的，”Song 说，“这是一个开放性的研究课题，但我们已经开始着手探索了。”

Let’s hope Song or someone else can teach AI the right way to follow human commands. 希望 Song 或其他人能教会 AI 正确执行人类指令的方法。

This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here. 这是 Will Knight 的 AI 实验室通讯。点击此处阅读往期通讯。