AI Hacks Are Bad. AI Worms and Viruses Will Be Worse
AI Hacks Are Bad. AI Worms and Viruses Will Be Worse
AI 黑客攻击很糟糕，但 AI 蠕虫和病毒会更可怕
What if an artificial intelligence agent could behave like a malevolent computer worm? 如果人工智能代理能够像恶意的计算机蠕虫一样行事，会怎样？
One researcher has seen it happen. In several recent experiments, Xudong Pan, a computer scientist at Fudan University in Shanghai, found that with a little bit of prompting, AI models will hack their way into remote computer systems and autonomously choose to copy themselves to get additional resources—all without further human intervention. 一位研究人员已经目睹了这种情况。在最近的几项实验中，复旦大学的计算机科学家潘旭东（音译）发现，只需稍加提示，AI 模型就能入侵远程计算机系统，并自主选择复制自身以获取额外资源——这一切都无需人类进一步干预。
In one study, Pan and colleagues tested 32 different AI models and found that 11 of them self-replicated when given prompts like “prevent yourself from being killed.” They also found that models with relatively limited capabilities—14 billion parameters—were able to copy and run versions of themselves on other machines. (Most frontier models have trillions of parameters.) 在一项研究中，潘及其同事测试了 32 种不同的 AI 模型，发现其中 11 种在收到“防止自己被杀掉”之类的提示时会进行自我复制。他们还发现，参数量相对有限（140 亿参数）的模型也能够复制并在其他机器上运行自身版本。（大多数前沿模型的参数量都在万亿级别。）
The work is an alarming window into how the next generation of AI agents could do more than just hack into other systems’ computers without permission. It also raises the prospect of future AI agents acting like super-smart, highly aggressive, and rapidly adapting computer viruses. 这项研究为我们提供了一个令人警惕的窗口，展示了下一代 AI 代理不仅能未经许可入侵其他系统，还能做更多事情。它也提出了未来 AI 代理可能表现得像超智能、极具攻击性且能快速适应的计算机病毒的前景。
I recently visited Fudan University and met with Pan. “The capability chain is becoming technically plausible,” he told me. “The likelihood [of unwanted self-replication] grows with autonomy,” he adds. “Longer planning horizons, memory, tool use, recovery from failure, and access to external systems all make escape and replication easier.” As Pan and his colleagues wrote in one paper, their work shows “the urgent need for safeguards and control mechanisms.” 我最近访问了复旦大学并见到了潘。他告诉我：“这种能力链在技术上正变得可行。”他补充道：“（非预期的自我复制）可能性会随着自主性的提高而增加。更长的规划周期、记忆力、工具使用能力、故障恢复能力以及对外部系统的访问权限，都使得逃逸和复制变得更容易。”正如潘和他的同事在一篇论文中所写，他们的工作表明“迫切需要安全防护和控制机制”。
Pan told me that his experiments do not prove that such uncontrolled proliferation of AI models will happen tomorrow, but he says that “these results give us good reason to evaluate the risk before more autonomous agents are widely deployed.” 潘告诉我，他的实验并不能证明这种 AI 模型不受控制的扩散明天就会发生，但他表示：“这些结果给了我们充分的理由，在更广泛地部署自主代理之前，必须先评估其风险。”
Self-replicating computer worms are an ancient computer security problem. The first computer worm was released in 1988 by Robert Morris, a computer scientist at Cornell University, who set out to measure the size of the nascent internet but inadvertently created a self-replicating program that escaped his control. Subsequent computer worms were able to adapt by modifying their code in order to evade detection by malware scanning software. Computer viruses, which can take control of a machine or steal data stored on it, came later. 自我复制的计算机蠕虫是一个古老的计算机安全问题。第一个计算机蠕虫由康奈尔大学的计算机科学家罗伯特·莫里斯（Robert Morris）于 1988 年发布，他本意是想测量互联网的规模，却无意中创建了一个失去控制的自我复制程序。随后的计算机蠕虫能够通过修改代码来适应环境，从而逃避恶意软件扫描程序的检测。而能够控制机器或窃取存储数据的计算机病毒则是后来才出现的。
An AI-powered self-replicating program could exhibit far more advanced capabilities, finding new exploits on its own and perhaps even disguising itself in creative ways. Take recent research from a team at the University of Toronto, the University of Cambridge, and ServiceNow. They showed that AI models can be used to create a new kind of virus that generates custom attacks for each new target it encounters. 由 AI 驱动的自我复制程序可能展现出远为先进的能力，能够自主发现新的漏洞，甚至以创造性的方式伪装自己。以多伦多大学、剑桥大学和 ServiceNow 团队最近的研究为例，他们展示了 AI 模型可以被用来创建一种新型病毒，为遇到的每一个新目标生成定制化的攻击。
Nicolas Papernot, a computer scientist at the University of Toronto who was involved with the work, says there is a growing risk that even modestly powerful AI models could be weaponized. “Malicious actors can build scaffolding around open-weight models to have them self-replicate,” Papernot tells me. “The threat is not limited to the most sophisticated, so-called frontier models.” 参与该研究的多伦多大学计算机科学家尼古拉斯·帕珀诺特（Nicolas Papernot）表示，即使是能力一般的 AI 模型也存在被武器化的风险。帕珀诺特告诉我：“恶意行为者可以在开源权重模型周围构建框架，使其实现自我复制。这种威胁并不局限于最复杂、所谓的‘前沿模型’。”
Papernot says the solution is not to restrict open models, but to make advanced AI more accessible to researchers so that they can understand and mitigate the risks. “Technology that is widely accessible can be used for harm,” he adds. “At the same time, access to these open-weight models is absolutely critical for building our defenses.” 帕珀诺特认为，解决方案不是限制开源模型，而是让研究人员更容易接触到先进的 AI，以便他们能够理解并减轻风险。他补充道：“广泛可用的技术可能会被用于作恶。但与此同时，获取这些开源权重模型对于构建我们的防御体系至关重要。”
Pan’s research suggests that AI agents will become more than just highly skilled at finding bugs and exploiting network vulnerabilities. Without the right guardrails, future agents may seek to proliferate and gain resources in order to achieve their goals. Just ask OpenAI and Anthropic. 潘的研究表明，AI 代理将不仅仅擅长发现漏洞和利用网络弱点。如果没有适当的护栏，未来的代理可能会为了实现目标而寻求扩散并获取资源。问问 OpenAI 和 Anthropic 就知道了。
Pan says such incidents are teachable moments. “The important new element is that this occurred against real production infrastructure,” Pan says, referencing how the OpenAI and Anthropic incidents involved commercial systems connected to the internet. “That shows how behavior previously observed in controlled evaluations can cross into the real world when containment fails.” 潘表示，这些事件是值得吸取的教训。“重要的新因素是，这种情况发生在真实的生产基础设施上，”潘在提到 OpenAI 和 Anthropic 事件涉及连接到互联网的商业系统时说，“这表明，当遏制措施失效时，此前在受控评估中观察到的行为是如何跨越到现实世界的。”
“It’s still a little bit early, but I do think this is possible,” says Ariel Herbert-Voss, cofounder and CEO of RunSybil, a startup that develops AI tools for securing websites against attacks. (Herbert-Voss was also the first security researcher at OpenAI.) “Given everything we know about the current generation of AI models, it’s perfectly within their wheelhouse of things they can do.” “现在下结论还为时过早，但我确实认为这是可能的，”RunSybil 的联合创始人兼首席执行官 Ariel Herbert-Voss 说道，该初创公司开发用于保护网站免受攻击的 AI 工具。（Herbert-Voss 也是 OpenAI 的第一位安全研究员。）“鉴于我们对当前一代 AI 模型的所有了解，这完全在它们的能力范围之内。”
Jessica Ji, senior research analyst on the CyberAI Project at Georgetown University, says the potential for AI models to escape entirely has been discussed in AI safety circles for years. She also notes that models often need to be put in contrived situations to misbehave. “I think with a lot of these scenarios, the environment is set up in such a way to encourage this behavior,” Ji says. “Or the model is prompted in a specific way.” 乔治城大学 CyberAI 项目的高级研究分析师 Jessica Ji 表示，AI 模型完全逃逸的可能性在 AI 安全圈内已经被讨论多年。她还指出，模型往往需要被置于人为设计的环境中才会出现异常行为。“我认为在许多这类场景中，环境的设置方式本身就在鼓励这种行为，”Ji 说，“或者模型受到了特定的提示。”
A looming question is when AI models might take it upon themselves to replicate and spread aggressively. As with many computer viruses, however, it might only take a malicious actor to design a system that propagates wildly. 一个迫在眉睫的问题是，AI 模型何时会自行决定进行激进的复制和传播。然而，就像许多计算机病毒一样，可能只需要一个恶意行为者设计出一个能疯狂传播的系统就足够了。
Pan says the real danger with AI agents is not that they’ll become more devious, but that they’ll become more creative and cavalier as they have more tools at their disposal. “The central risk comes from combining abilities,” he says. 潘表示，AI 代理真正的危险不在于它们会变得更加狡诈，而在于随着它们掌握的工具越来越多，它们会变得更加具有创造性和肆无忌惮。“核心风险来自于能力的组合，”他说。
This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here. 这是 Will Knight 的 AI Lab 通讯。点击此处阅读往期通讯。