The specter of AI-enabled bioweapons is a wake-up call for biotech

AI 生物武器的阴影：生物技术行业的一记警钟

EXECUTIVE SUMMARY In recent weeks, leaders of some of the biggest AI companies have warned that the very tech they are developing is dangerous. Last weekend, Anthropic CEO Dario Amodei argued that AI carries serious risk and that progress should be slowed. OpenAI CEO Sam Altman responded on X: “I agree with Dario that we need to pace the frontier.” 执行摘要 最近几周，一些顶级人工智能公司的领导人发出警告，称他们正在开发的技术本身就存在危险。上周末，Anthropic 首席执行官 Dario Amodei 指出，人工智能带来了严重的风险，其发展速度应当放缓。OpenAI 首席执行官 Sam Altman 在 X（原推特）上回应称：“我同意 Dario 的观点，我们需要控制前沿技术的发展节奏。”

Those posts came a few days after the AI researcher Jacob Coxon announced that he was leaving a role at Anthropic, charging that neither it nor OpenAI (where he had also worked) was acting responsibly. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he posted on X. Another Anthropic employee, Evan Hubinger, publicly agreed with him. “We really do earnestly believe AI could kill all humans!” he responded on X. “I personally think it is >10% within the next decade.” 这些言论发布的前几天，人工智能研究员 Jacob Coxon 宣布辞去在 Anthropic 的职务，并指责该公司及其曾任职的 OpenAI 均未采取负责任的行动。他在 X 上写道：“那些构建人工智能的人真诚地相信，到本十年末，它可能会杀死我们所有人。”另一位 Anthropic 员工 Evan Hubinger 公开表示赞同。他在 X 上回应道：“我们确实真诚地相信人工智能可能会杀死全人类！我个人认为在未来十年内发生的概率超过 10%。”

One of the ways they fear AI might end us all is by somehow aiding the design, creation, and release of some kind of bioweapon. Let’s take a closer look at why. A bioweapon might be a highly lethal virus that targets people according to their genes. It could be a fungus that wipes out a crop and causes food insecurity. Perhaps it would be a tasteless, odorless toxin that could be slipped into a region’s water supply, undetected. 他们担心人工智能终结人类的方式之一，是它可能以某种方式协助设计、制造并释放某种生物武器。让我们深入探讨一下原因。生物武器可能是一种根据基因靶向人类的高致死性病毒；也可能是一种摧毁农作物并导致粮食危机的真菌；又或许是一种无色无味的毒素，可以神不知鬼不觉地投放到某个地区的供水系统中。

The concern is that AI tools can be used to help generate agents like these. In 2022, researchers at Collaborations Pharmaceuticals found that it was remarkably easy to do so using an AI “molecule generator” they’d developed to find potential drugs for human disease. In less than six小时, the model generated 40,000 molecules with the potential to serve as chemical warfare agents. Some of them were designed to be even more toxic than known nerve agents. 令人担忧的是，人工智能工具可以被用来帮助生成这类制剂。2022 年，Collaborations Pharmaceuticals 的研究人员发现，利用他们为寻找人类疾病潜在药物而开发的人工智能“分子生成器”，可以非常容易地做到这一点。在不到六小时的时间里，该模型生成了 4 万种具有化学战剂潜力的分子。其中一些分子的设计毒性甚至超过了已知的神经毒剂。

“Without being overly alarmist, this should serve as a wake-up call for our colleagues in the ‘AI in drug discovery’ community,” the authors wrote at the time. It was a wake-up call for David Magnus, a professor of medicine and biomedical ethics at Stanford University, even though he had been assessing the risks associated with the misuse of medical science and biotechnology since the late 1990s. “That was very scary to me,” he says. “Of course, everything since then has just sort of blown up.” 当时的作者写道：“我们并非危言耸听，但这应该为‘人工智能药物研发’社区的同行们敲响警钟。”对于斯坦福大学医学与生物医学伦理学教授 David Magnus 来说，这确实是一记警钟，尽管他自 20 世纪 90 年代末以来就一直在评估滥用医学科学和生物技术的风险。他说：“那让我感到非常恐惧。当然，从那以后，一切都呈现出爆发式增长。”

Today, AI bots can answer questions on topics spanning all realms of science. Anyone can use large language models trained on the knowledge and experience of “almost every scientist who ever lived on this planet,” says Dunja Sabra, a biosecurity researcher at the University of Hamburg in Germany. Those models can provide instructions and video training on how to conduct experiments. Combine that with advances in biotech that have made gene editing and synthetic biology tools much more accessible (the “DIY biology” movement has already enabled many people to set up labs at home), and you’ve got a potentially very dangerous situation. “The chances are that someone determined would succeed eventually,” Sabra says. 如今，人工智能机器人可以回答涵盖所有科学领域的问题。德国汉堡大学的生物安全研究员 Dunja Sabra 表示，任何人都可以使用基于“地球上几乎每一位曾经存在过的科学家”的知识和经验训练出来的大型语言模型。这些模型可以提供如何进行实验的说明和视频培训。再加上生物技术的进步使得基因编辑和合成生物学工具变得更加普及（“DIY 生物学”运动已经使许多人能够在家里建立实验室），这就形成了一个潜在的非常危险的局面。Sabra 说：“很有可能，意志坚定的人最终会成功。”

There are safeguards in place. People who want to build new genomes must typically order the pieces of DNA from companies that screen for suspicious requests. Responsible researchers put potentially risky research through rounds of analysis called “red-teaming,” in which independent scientists look for ways the work might be misused, and “blue-teaming,” where others come up with potential mitigations. And AI companies have tweaked their tools in attempts to prevent them from offering up scientific information that could be misused. 目前已有相应的保障措施。想要构建新基因组的人通常必须从会对可疑请求进行筛查的公司订购 DNA 片段。负责任的研究人员会对潜在的风险研究进行多轮分析，即“红队测试”（由独立科学家寻找该工作可能被滥用的方式）和“蓝队测试”（由其他人提出潜在的缓解措施）。此外，人工智能公司也对其工具进行了调整，试图防止它们提供可能被滥用的科学信息。

But none of these protections are ironclad. In a report published last week, Anthropic acknowledged that people had attempted to use its models to explore ways to make the chikungunya virus more transmissible, create a form of bird flu that is more dangerous to humans, and build an “atlas of venom toxin peptides,” among other things. “We’ve got a constant back and forth,” says Magnus. “We have to build better surveillance and screening tools, [but] AI is really good at figuring out ways around them.” We’ll probably need to use AI to find ways to restrict the use of AI, he says. 但这些保护措施并非无懈可击。在上一周发布的一份报告中，Anthropic 承认有人曾试图利用其模型探索如何使基孔肯雅病毒更具传染性、制造一种对人类更危险的禽流感病毒，以及构建“毒素肽图谱”等。Magnus 说：“我们正处于不断的博弈中。我们必须建立更好的监控和筛查工具，但人工智能非常擅长寻找绕过这些工具的方法。”他表示，我们可能需要利用人工智能来寻找限制人工智能使用的方法。

I should add here that not all scientists agree on the level of risk. At a recent media briefing, some biologists at Imperial College London argued that AI tools just aren’t good enough to fully develop bioweapons, and that testing new pathogens requires difficult, time-consuming, human work. Some think the guardrails we have in place are sufficient. And Wendy Barclay, a professor of infectious disease at Imperial, pointed out that, as things stand, the greatest risk of a pandemic isn’t from a bioweapon, but from pathogens that are already circulating. Take H5N1, the bird flu virus that has already killed millions of birds and spread widely through US dairy cattle; last month it was also detected in captive mink at a farm in Utah. 我需要补充的是，并非所有科学家都认同风险的程度。在最近的一次媒体吹风会上，伦敦帝国理工学院的一些生物学家认为，人工智能工具目前还不足以完全开发出生物武器，而且测试新病原体需要困难、耗时的人工操作。有些人认为现有的护栏已经足够。帝国理工学院传染病学教授 Wendy Barclay 指出，就目前情况而言，大流行的最大风险并非来自生物武器，而是来自已经在传播的病原体。以 H5N1 为例，这种禽流感病毒已经杀死了数百万只鸟类，并在美国奶牛群中广泛传播；上个月，它还在犹他州一家农场的圈养水貂身上被检测到。

Sabra, on the other hand, likes to think five to 10 years ahead. Countries should be strengthening their health-care systems, preparing antidotes to known toxins, and stockpiling medicines, she says: “We need to be prepared.” Kevin Esvelt, an MIT biologist who invented both technology to fast-track the propagation of a genetic feature through an entire population and ways to limit that technology, echoed these concerns in an X post on Wednesday, stating that a large language model had “disclosed a novel form of bioweapon that I hadn’t realized was possible.” He added, “Please, for the love of God, children, the future of humanity, or whatever you consider holy, let’s err on the side of caution here.” 另一方面，Sabra 倾向于从未来 5 到 10 年的角度思考问题。她说，各国应该加强医疗保健系统，准备已知毒素的解毒剂，并储备药物：“我们需要做好准备。”麻省理工学院生物学家 Kevin Esvelt（他发明了加速基因特征在整个种群中传播的技术，同时也发明了限制该技术的方法）周三在 X 上发帖呼应了这些担忧，称一个大型语言模型“揭示了一种我此前未曾意识到可能存在的新型生物武器”。他补充道：“求求你们，为了上帝、为了孩子、为了人类的未来，或者任何你们认为神圣的事物，让我们在这一点上保持谨慎。”