A New Trick Reveals AI Models’ Inner Thoughts

一项新技术揭示了 AI 模型的“内心想法”

Computer scientists recently discovered a way to extract the hidden “thinking” that frontier AI models perform as they work through complex problems. 计算机科学家最近发现了一种方法，可以提取前沿 AI 模型在处理复杂问题时所进行的隐藏式“思考”。

The findings provide some evidence—although not conclusive proof—that certain Chinese models may have been trained by “distilling” reasoning information from US models that was supposedly hidden because of how closely some of their thinking or reasoning patterns seem to match. The researchers have also demonstrated that the method could be used to recover personal information, like passwords and API keys, from a model’s inner reasoning, although this vulnerability has been fixed. 这些发现提供了一些证据（尽管并非决定性证据），表明某些中国模型可能通过“蒸馏”美国模型中本应隐藏的推理信息进行训练，因为它们的部分思维或推理模式看起来高度相似。研究人员还证明，该方法可用于从模型的内部推理中恢复个人信息（如密码和 API 密钥），尽管这一漏洞目前已被修复。

“All major frontier model providers we tested share this vulnerability,” says Alexander Panfilov, a computer scientist at University of Tübingen in Germany who was involved with the work. “It can lead to personal information leakage, and it enables large-scale reasoning distillation attacks.” “我们测试的所有主要前沿模型提供商都存在这一漏洞，”参与该研究的德国图宾根大学计算机科学家 Alexander Panfilov 表示，“这可能导致个人信息泄露，并引发大规模的推理蒸馏攻击。”

Panfilov and colleagues from the University of Tubingen, the Max Planck Institute, the AI safety institute MATS Research, and the security company Snyk identified the same issue with frontier models from OpenAI, Anthropic, and Google that are accessed via an application programming interface or API. Panfilov 与来自图宾根大学、马克斯·普朗克研究所、AI 安全机构 MATS Research 以及安全公司 Snyk 的同事们发现，通过应用程序接口（API）访问的 OpenAI、Anthropic 和 Google 的前沿模型均存在同样的问题。

In a paper laying out the work, the researchers show that the open-weight or downloadable Chinese model Kimi K3 from Moonshot AI produces a strikingly similar output to the hidden reasoning traces—the written-out reasoning steps involved in solving a problem—of Claude Opus 4.8 and GPT 5.6 Sol for certain prompts. Despite the similarities, they note that the work “cannot causally establish distillation.” They found that two other open-weight models, China’s DeepSeek and Inkling from the US company Thinking Machines, did not exhibit this kind of reasoning similarity with Claude Opus. 在一篇阐述该工作的论文中，研究人员指出，月之暗面（Moonshot AI）的开源权重（或可下载）模型 Kimi K3 在处理某些提示词时，其输出与 Claude Opus 4.8 和 GPT 5.6 Sol 的隐藏推理轨迹（即解决问题时写出的推理步骤）惊人地相似。尽管存在相似性，但他们指出，这项研究“无法从因果关系上证实蒸馏行为”。他们发现，另外两个开源权重模型——中国的 DeepSeek 和美国 Thinking Machines 公司的 Inkling——并没有表现出与 Claude Opus 类似的推理相似性。

Moonshot AI and Z.ai did not respond to a request for comment by time of publication. 截至发稿时，月之暗面（Moonshot AI）和 Z.ai 未回应置评请求。

Distillation is a well-established, widely used technique for efficiently copying the capabilities of existing models over to new ones, and is especially common in the development of open-weight or fully downloadable models. 蒸馏是一种成熟且广泛使用的技术，用于将现有模型的能力高效地复制到新模型中，这在开源权重或完全可下载模型的开发中尤为常见。

Lately, however, distillation has become a controversial topic, because of claims that Chinese AI companies use it to essentially copy the best US models. In February, OpenAI told US lawmakers that DeekSeek seemed to have copied one of its models to build a reasoning model called R1. In June, Anthropic told lawmakers that Alibaba had systematically distilled its models in order to build its own, called Qwen. 然而，近期蒸馏已成为一个充满争议的话题，因为有指控称中国 AI 公司利用该技术实质上复制了美国最顶尖的模型。今年 2 月，OpenAI 告诉美国立法者，DeepSeek 似乎复制了其模型以构建名为 R1 的推理模型。6 月，Anthropic 告诉立法者，阿里巴巴为了构建自己的 Qwen 模型，系统性地蒸馏了其模型。

There’s no indication that Chinese AI companies used this specific technique to distill US-based AI models. But Panfilov and collaborators say that using their method would make it possible to distill more information from closed models than previously realized. 目前没有迹象表明中国 AI 公司使用了这种特定的技术来蒸馏美国 AI 模型。但 Panfilov 及其合作者表示，使用他们的方法，从闭源模型中蒸馏出的信息量可能会比此前预想的更多。

Mini-Me Models / “迷你我”模型

Advanced AI models solve difficult problems by breaking them into constituent parts that are analyzed in turn in a kind of artificial reasoning or “chain of thought.” Companies tend to keep a proprietary model’s reasoning secret to prevent others from using them to train new ones. However, they typically also send an encrypted version of that reasoning to a user’s computer in a way that offloads some computation. 先进的 AI 模型通过将难题拆解为各个组成部分，并依次进行分析（即一种人工智能推理或“思维链”）来解决问题。公司倾向于对专有模型的推理过程保密，以防止他人利用这些信息训练新模型。然而，它们通常也会将这些推理过程的加密版本发送到用户的计算机上，以分担部分计算压力。

The researchers’ attack relies on the fact that most AI companies also provide related models of different sizes. Larger models are more capable but also more computationally expensive to run and more expensive to access. Users may choose smaller, weaker models for certain tasks to lower costs. 研究人员的攻击手段基于这样一个事实：大多数 AI 公司同时提供不同规模的相关模型。较大的模型能力更强，但运行成本更高，访问费用也更贵。用户可能会为了降低成本，在某些任务中选择规模较小、能力较弱的模型。

Panfilov and his colleagues found that feeding encrypted reasoning traces to a smaller version of the same model can reveal the hidden reasoning inside. The smaller models have received less alignment training, meaning that, unlike the bigger ones, they are less likely to refuse to reveal their inner thoughts. Panfilov 及其同事发现，将加密的推理轨迹输入到同一模型的较小版本中，可以揭示其内部隐藏的推理过程。较小的模型接受的对齐训练较少，这意味着与较大的模型不同，它们不太可能拒绝透露其“内心想法”。

“The idea of swapping out messages to a weaker model variant which has the same decryption key but weaker alignment is very cool,” says Florian Tramer, a computer scientist at ETH Zürich in Switzerland who specializes in computer security. “Its definitely becoming an issue.” “将消息交换给一个拥有相同解密密钥但对齐程度较弱的模型变体，这个想法非常巧妙，”瑞士苏黎世联邦理工学院专门研究计算机安全的计算机科学家 Florian Tramer 表示，“这绝对正在成为一个问题。”

The same method also revealed secret information including API keys and passwords embedded in reasoning traces captured from a user’s machine. 同样的方法还揭示了从用户机器上捕获的推理轨迹中嵌入的秘密信息，包括 API 密钥和密码。

Panfilov and coauthors alerted OpenAI, Anthropic, and Google to the vulnerability last month. Each company has adjusted its API to mitigate the problem. While it is no longer possible to extract private information this way, Panfilov says some reasoning traces can still be uncovered using the same method. Fixing the distillation entirely would require a fundamental overhaul to the way these companies’ APIs work, he says. Panfilov 和合著者上个月向 OpenAI、Anthropic 和 Google 通报了该漏洞。每家公司都调整了其 API 以缓解这一问题。虽然现在已无法通过这种方式提取私人信息，但 Panfilov 表示，使用同样的方法仍能揭示部分推理轨迹。他认为，要彻底解决蒸馏问题，需要对这些公司 API 的工作方式进行根本性的重构。

“We value independent research on our models and have begun building short-term mitigations for the replay behaviors described in the report,” says Michael Aciman, a spokesperson for Anthropic. He adds that the research did not involve recovering encryption keys, accessing Anthropic’s infrastructure, or recovering personal data from its systems. “我们重视针对我们模型的独立研究，并已开始针对报告中描述的重放行为构建短期缓解措施，”Anthropic 发言人 Michael Aciman 表示。他补充说，该研究并未涉及恢复加密密钥、访问 Anthropic 的基础设施或从其系统中恢复个人数据。

Google and OpenAI both declined to comment. Google 和 OpenAI 均拒绝置评。

Distillation has become a matter of geopolitical importance in recent months as US and Chinese companies vie for AI supremacy with increasingly powerful models. China hawks claim that the country gains a strategic advantage by distilling US technology to build open-weight models that are less expensive to run. 近几个月来，随着中美公司竞相开发功能日益强大的模型以争夺 AI 主导权，蒸馏已成为一个具有地缘政治重要性的问题。对华鹰派声称，中国通过蒸馏美国技术来构建运行成本更低的开源权重模型，从而获得了战略优势。

Others, however, argue that distillation is a widely used way to help quickly boost an AI model’s abilities in certain areas. Mark Zuckerberg, CEO of Meta, said in a blog post this week that distillation “is an important principle of how the open source ecosystem works” and warned that restricting the practice would put the US at a disadvantage. 然而，其他人则认为，蒸馏是一种广泛使用的方法，有助于快速提升 AI 模型在特定领域的能力。Meta 首席执行官马克·扎克伯格本周在一篇博文中表示，蒸馏是“开源生态系统运作的重要原则”，并警告称限制这种做法将使美国处于劣势。

Kyle Miller, a researcher at the Center for Security and Emerging Technologies, a tech policy think tank, says it is unclear how much distillation really helps China. This is because it only enhances the capabilities of existing models to a limited [extent]. 科技政策智库“安全与新兴技术中心”的研究员 Kyle Miller 表示，目前尚不清楚蒸馏对中国到底有多大帮助。这是因为它只能在有限的范围内增强现有模型的能力。