US Military had close call after using AI for hallucinated intelligence report
US Military had close call after using AI for hallucinated intelligence report
美军因使用人工智能生成的虚假情报险些酿成大祸
The intelligence report, circulated across the US military this spring in the midst of the war with Iran, immediately set off alarm bells: A Chinese ship in the Middle East was transporting components of a nuclear weapons program. 今年春天，在与伊朗的战争期间，一份在美国军方内部流传的情报报告立即敲响了警钟：一艘在中东地区的中国船只正在运输核武器项目的组件。
The US military swung into action with plans to intercept the vessel, according to four sources familiar with the episode. According to two of the sources, armed members of the US military were preparing to board the ship. Military planes were in the air, one of those sources and another source familiar with the incident said. 据四位知情人士透露，美军随即采取行动，计划拦截该船只。其中两位消息人士称，美军武装人员当时正准备登船。另一位知情人士和一名了解该事件的消息来源表示，当时军用飞机已经升空。
It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. CNN was not able to learn what the misidentified cargo was. 直到计划行动前夕，官员们才对这份由特种作战司令部分析师撰写的报告进行了深入调查，发现该报告是在人工智能（AI）的帮助下生成的，且分析师所使用的聊天机器人错误地识别了船上运载的物资。CNN未能获悉被误认的货物究竟是什么。
The report, according to one of the sources, was “entirely false.” But it also “almost started a war,” the source said. Any US operation against a Chinese vessel could have risked spiraling into an armed conflict between the two nations. 据其中一位消息人士称，这份报告“完全是虚假的”。但该消息人士表示，它也“差点引发一场战争”。任何针对中国船只的美国行动，都有可能导致两国之间爆发武装冲突。
Across the US military and the intelligence community, officials are pushing to weave AI into nearly every facet of their work, from analyzing the huge volumes of raw intelligence the US collects and selecting targets for strikes, to more mundane applications like managing budgeting, logistics and supply chains. 在美国军方和情报界，官员们正推动将人工智能融入其工作的几乎每一个方面，从分析美国收集的海量原始情报并选择打击目标，到管理预算、物流和供应链等更为常规的应用。
But the episode underscores the profound risks of using this powerful, new and relatively poorly understood technology for targeting in the middle of a war. Analysts have long feared that AI could lead to a catastrophic miscalculation if nation states are relying on poor or corrupted data — the kind of miscalculation that might lead the United States to fire on a Chinese ship based on inaccurate information. 然而，这一事件凸显了在战争期间使用这种强大、新兴且相对缺乏了解的技术进行目标定位所带来的深重风险。分析人士长期以来一直担心，如果国家依赖糟糕或被篡改的数据，人工智能可能会导致灾难性的误判——这种误判可能导致美国基于不准确的信息向中国船只开火。
In this particular instance, the analyst queried a chatbot about some intelligence reporting on the ship’s manifest that originated with US Special Operations Command Pacific, based in Hawaii. It was not clear whether the chatbot was a commercially available one or a US government product. 在这次特定事件中，该分析师向一个聊天机器人查询了有关该船只舱单的情报报告，这些情报源自位于夏威夷的美国太平洋特种作战司令部。目前尚不清楚该聊天机器人是市面上的商业产品，还是美国政府内部的产品。
“The internal tools are mostly just copies of the commercial stuff wearing lipstick,” a former senior US official familiar with the AI systems used by military and intelligence analysts. The bot fused together open-source intelligence with secret signals intelligence in government holdings and reached its fateful conclusion about the material the ship was carrying. 一位熟悉军方和情报分析师所用AI系统的前美国高级官员表示：“内部工具大多只是给商业产品‘涂脂抹粉’后的复制品。”该机器人将开源情报与政府持有的秘密信号情报融合在一起，并就船上运载的物资得出了那个致命的结论。
The analyst then used AI again to package the findings into a standard intelligence report — the kind that is trusted by military officials — and disseminated it. US Special Operations Command Pacific and the Pentagon did not respond to a request for comment. 随后，该分析师再次利用人工智能将这些发现打包成一份标准的情报报告——即那种受军方官员信任的报告——并进行了分发。美国太平洋特种作战司令部和五角大楼未回应置评请求。
The rationale for the rapid adoption of AI is that it can help the military make battlefield decisions, like which targets to strike or which military assets to move where, faster. Officials say the US can’t afford to fall behind in integrating AI in case it must one day fight China or another adversary who would potentially be able to stay one step ahead of the US. 快速采用人工智能的理由是，它可以帮助军方更快地做出战场决策，例如打击哪些目标或将哪些军事资产部署到何处。官员们表示，美国在整合人工智能方面不能落后，以防有朝一日必须与中国或其他可能领先美国一步的对手作战。
In January, Defense Secretary Pete Hegseth released his agency’s “Artificial Intelligence Acceleration Strategy” in a bid to speed up the military’s use of AI. “We will unleash experimentation, eliminate bureaucratic barriers, focus our investments and demonstrate the execution approach needed to ensure we lead in military AI,” Hegseth said in a speech announcing the strategy. 今年1月，国防部长皮特·赫格塞斯（Pete Hegseth）发布了该机构的“人工智能加速战略”，旨在加快军方对人工智能的使用。赫格塞斯在宣布该战略的演讲中表示：“我们将释放实验潜力，消除官僚障碍，集中投资，并展示确保我们在军事人工智能领域保持领先地位所需的执行方法。”
The strategy also pushes for its broad use across the military, ordering the department to make AI available via several programs with the aim of “democratizing AI experimentation and transformation across the Department by putting America’s world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels,” a memo announcing the strategy said. 该战略还推动人工智能在全军的广泛使用，要求国防部通过多个项目提供人工智能，旨在“通过将美国世界领先的人工智能模型直接交到我们三百万文职和军事人员手中，实现国防部内部人工智能实验和转型的民主化，并覆盖所有密级水平，”一份宣布该战略的备忘录写道。
But the effort is decentralized, multiple US officials familiar with the dynamic said, with different parts of the government using different tools under different orders and safety standards. There’s no one set of standards for how the US verifies the information generated by these tools. 但多位熟悉此动态的美国官员表示，这项工作是分散进行的，政府的不同部门在不同的指令和安全标准下使用不同的工具。对于美国如何验证这些工具生成的信息，目前尚无统一的标准。
The constellation of different AI systems being deployed by disparate corners of the military and intelligence community means that the relative reliability and functionality vary widely. For weeks, Washington policymakers have been intensely debating AI after a series of dire warnings from Silicon Valley engineers and tech CEOs of the possibility that AI could break free of human constraints, with potentially civilization-ending consequences. 军方和情报界不同部门部署的各种人工智能系统意味着其相对可靠性和功能性差异巨大。几周以来，在硅谷工程师和科技公司首席执行官发出了一系列严厉警告，称人工智能可能摆脱人类束缚并带来可能终结文明的后果后，华盛顿的决策者们一直在激烈辩论人工智能问题。
But the episode with the Chinese ship underscores a different, and more immediate risk: human beings making disastrous decisions based on inaccurate or misleading information generated by AI or other automated systems. Sources said that the military is rapidly turning to AI to help with targeting, an area which holds the obvious risk of fatal mistakes. 但中国船只事件凸显了另一种更直接的风险：人类基于人工智能或其他自动化系统生成的错误或误导性信息，做出灾难性的决策。消息人士称，军方正迅速转向利用人工智能协助目标定位，而这一领域存在明显的致命错误风险。
“AI in targeting is definitely something that is ramping up and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide,” another source familiar with the military’s current policies said. 另一位熟悉军方现行政策的消息人士表示：“人工智能在目标定位中的应用确实在增加，但对于如何通过‘人在回路’（human in the loop）来防止平民伤亡或误伤友军，目前还没有真正的指导方针。”
The kind of “hallucination” that the tool used by the analyst in this case conjured has not been an isolated incident across the intelligence community since these tools began proliferating across government, according to one of the sources. 据其中一位消息人士称，自这些工具开始在政府内部普及以来，该分析师所使用的工具所产生的这种“幻觉”在情报界并非孤立事件。