AI Hallucination Nearly Triggers US Military Operation

AI“幻觉”险些引发美国军事行动

Military aircraft were already in the air this spring when U.S. officials made an alarming discovery: the intelligence driving an armed operation against a Chinese vessel had been hallucinated by an AI chatbot. The operation was aborted at the last minute, narrowly averting a potential conflict with China, CNN reported on Friday. 今年春天，当美国军方官员发现一个惊人的事实时，军用飞机已经升空：针对一艘中国船只采取武装行动的情报，竟是由人工智能聊天机器人“幻觉”产生的。据美国有线电视新闻网（CNN）周五报道，该行动在最后一刻被取消，从而险些避免了一场与中国的潜在冲突。

The episode underscores a growing concern among military officials and outside experts: As decision-makers lean more heavily on AI, the errors these systems produce can travel up the chain of command before being questioned. The intelligence report, which circulated during the war with Iran, said the vessel was carrying components for a nuclear weapons program. 这一事件凸显了军事官员和外部专家日益增长的担忧：随着决策者越来越依赖人工智能，这些系统产生的错误可能会在受到质疑之前就层层上报至指挥链。这份在与伊朗冲突期间流传的情报报告称，该船只载有核武器计划的零部件。

The false intelligence originated with a Special Operations Command analyst who queried an AI chatbot to synthesize open-source data with classified signals intelligence. The chatbot misidentified the ship’s cargo manifest. The analyst then used the tool a second time to format the erroneous findings into an official-looking summary, which was circulated across command channels. 虚假情报源于特种作战司令部的一名分析师，他曾询问人工智能聊天机器人，要求将开源数据与机密信号情报进行综合分析。聊天机器人错误地识别了该船的货物清单。随后，该分析师第二次使用该工具，将错误的调查结果格式化为一份看起来很正式的摘要，并在指挥渠道中进行了传阅。

The near-miss comes as the U.S military races to integrate AI to accelerate decision-making and maintain its edge over China. The Pentagon has described AI as delivering a significant advantage in speeding up its kill chain so commanders can respond in the right time. But the same speed that makes AI attractive may also allow hallucinations with insufficient human oversight. 这次险情发生之际，美国军方正竞相整合人工智能，以加速决策并保持对中国的优势。五角大楼曾表示，人工智能在加速“杀伤链”（kill chain）方面具有显著优势，使指挥官能够及时做出反应。但人工智能所具备的这种速度优势，在缺乏足够人工监督的情况下，也可能导致“幻觉”的产生。

“It’s important for service members to understand the uncertainty inherent to LLMs,” said Jake Steckler, research scholar at GovAI and veteran U.S. Army officer in a written response to TechCrunch. “But it’s especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions.” “军方人员了解大语言模型（LLM）固有的不确定性非常重要，”GovAI研究学者、美国陆军退伍军官杰克·斯特克勒（Jake Steckler）在给TechCrunch的书面回复中表示。“但对于任何可能导致使用武力的决策，如目标锁定、情报分析或作战规划，这一点尤为关键。这些决策关乎生死。”

Still, Steckler says, the incident should serve as a call to add more safeguards to AI, not a reason to avoid it. “These tools can be useful in the right contexts and with the right safeguards in place,” he said. “But prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems, which ultimately is only going to slow adoption.” 尽管如此，斯特克勒认为，这一事件应该成为加强人工智能安全保障的警钟，而不是回避它的理由。“在合适的背景下并采取适当的保障措施，这些工具是非常有用的，”他说。“但如果将采用速度置于一切之上，很可能会导致一些事件发生，这只会让军方人员对这些系统失去信任，最终反而会减缓人工智能的普及。”