Anthropic found a hidden space where Claude puzzles over concepts

Anthropic 发现了一个隐藏空间，Claude 在其中推敲概念

The AI firm Anthropic has developed a technique that has given it the clearest glimpse yet at what’s really going on inside large language models as they answer questions or carry out tasks. What they found ranges from the mundane to the unnerving. 人工智能公司 Anthropic 开发出一种新技术，使其能够最清晰地洞察大型语言模型在回答问题或执行任务时内部究竟发生了什么。他们的发现从平淡无奇到令人不安，应有尽有。

Researchers at the company built a tool called the Jacobian lens (or J-lens) and used it to uncover a hidden area, which they named the J-space, inside Claude Opus 4.6, a version of Anthropic’s flagship LLM released in February. The J-space contains individual words that are related to the words and phrases that the model is most likely to spit out in a response in the near future. 该公司的研究人员构建了一种名为“雅可比透镜”（Jacobian lens，简称 J-lens）的工具，并利用它在 Claude Opus 4.6（Anthropic 于 2 月发布的旗舰大模型版本）内部发现了一个隐藏区域，他们将其命名为“J-空间”（J-space）。J-空间包含了一些独立的词汇，这些词汇与模型在不久的将来最可能输出的词语和短语密切相关。

If Claude were a person (which it is not), you might say that these hidden words can reveal what’s on its mind before it actually speaks. Anthropic found that what an LLM is actually doing can often be different from what it says it is doing. The company claims that monitoring words that pop up in the J-space gives it a new way to understand and control its models. 如果 Claude 是一个人（当然它不是），你可能会说这些隐藏的词汇揭示了它在开口说话前的心中所想。Anthropic 发现，大语言模型实际执行的操作往往与它声称的操作有所不同。该公司声称，监测 J-空间中出现的词汇为理解和控制模型提供了一种新途径。

The company shared its results in a paper posted on its website this week. It has also teamed up with Neuronpedia, an open-source platform that lets you poke around inside LLMs yourself, to make a hands-on demo that anyone can try. “It’s very good and interesting work,” says Tom McGrath, chief scientist and cofounder at Goodfire, a startup that also builds tools to understand and control LLMs. 该公司本周在其网站上发布的一篇论文中分享了研究结果。他们还与开源平台 Neuronpedia 合作，制作了一个任何人都可以尝试的实操演示，该平台允许用户亲自探索大语言模型的内部结构。同样致力于构建大语言模型理解与控制工具的初创公司 Goodfire，其首席科学家兼联合创始人 Tom McGrath 表示：“这是一项非常出色且有趣的工作。”

Going deeper

深入探究

For the last couple of years, Anthropic has been pushing the envelope in a field of research known as mechanistic interpretability, which involves probing the internal workings of LLMs to see how they tick. (MIT Technology Review picked mechanistic interpretability as one of this year’s top breakthrough technologies.) The new technique builds on previous work from Anthropic and others to expose a deeper level inside LLMs that researchers had not seen before. 过去几年里，Anthropic 一直在“机械可解释性”（mechanistic interpretability）这一研究领域不断突破，该领域旨在探测大语言模型的内部运作机制，以了解其运行原理。（《麻省理工科技评论》将机械可解释性评为今年的十大突破性技术之一。）这项新技术建立在 Anthropic 及其他机构过往研究的基础上，揭示了大语言模型内部更深层次的结构，这是研究人员此前从未见过的。

Picture an LLM as a stack of books. Each book is a layer of basic computational units known as neurons, with each neuron in one layer passing information to the neurons in the layers above. The books at the bottom of the stack are the input layers, which process the text coming into the model. The books at the top are the output layers, which prepare the text that the model is about to produce. Much of what goes on in these input and output layers is housekeeping. But in the middle of the stack, you get the layers that do the heavy lifting, churning through the complex math that turns prompts into responses one word at a time. That’s where the really clever—and mysterious—stuff happens. 将大语言模型想象成一叠书。每一本书都是一层基本的计算单元，即“神经元”，每一层的神经元都会将信息传递给上一层的神经元。叠在底部的书是输入层，负责处理进入模型的文本；顶部的书是输出层，负责准备模型即将生成的文本。输入层和输出层的大部分工作都是基础性的维护。但在书堆中间，才是真正承担繁重任务的层级，它们通过复杂的数学运算，将提示词逐字转化为回复。那里才是真正聪明——也最神秘——的地方。

To peer deeper into those middle layers, Anthropic adapted an existing tool called a logit lens. A logit lens can be used to look inside an LLM to identify the words that it is likely to produce next. Moving the lens down the stack of books reveals what words the LLM is focusing on at that particular point in its number crunching. Anthropic’s J-lens works in a similar way but picks out words that an LLM is likely to say at some point in the near future, not necessarily straight away. 为了深入观察这些中间层，Anthropic 改良了一种名为“Logit 透镜”（logit lens）的现有工具。Logit 透镜可以深入大语言模型内部，识别其下一步最可能生成的词汇。将透镜在书堆中上下移动，就能揭示模型在数值计算的特定阶段正关注哪些词汇。Anthropic 的 J-lens 工作原理类似，但它挑选的是模型在不久的将来某个时刻可能说出的词，而不一定是紧接着要说的词。

What that reveals in practice are words that are related to the response an LLM is working on but that might not actually end up being part of that response by the time the math in the middle layers has run its course. “When a model is operating, it’s not only trying to predict the next token,” says McGrath. “It’s also computing a lot of other things that might be useful for tokens that happen in the future.” Again, if Claude were a person (it’s not), you might say that the J-lens gives clues about what it is thinking about at different levels of the book stack but not saying out loud. 实际上，它揭示的是与模型正在处理的回复相关的词汇，但当中间层的数学运算完成后，这些词汇最终可能并不会出现在回复中。McGrath 说：“当模型运行时，它不仅在尝试预测下一个标记（token），还在计算许多其他可能对未来标记有用的信息。”再次强调，如果 Claude 是一个人（它不是），你可能会说 J-lens 提供了关于它在书堆不同层级中思考、但未说出口的内容的线索。

Stranger things

更离奇的事

“A lot of the time the contents of the J-space are fairly mundane,” says McGrath, who has tried out Anthropic’s J-lens himself. “But sometimes it produces quite surprising things that seem to be, like, sort of internal themes or thought processes.” “很多时候，J-空间里的内容相当平淡，”亲自尝试过 Anthropic J-lens 的 McGrath 说，“但有时它会产生一些非常令人惊讶的内容，看起来就像是某种内在的主题或思维过程。”

Anthropic gives a number of examples of what it found. Sometimes the J-lens exposed the steps that Claude took when it was working through a problem. For example, when it was asked to calculate (4+7)2+7, its J-space contained the word “math” and numbers representing the intermediate results “21” (for 4+7) and “42” (for 212). Anthropic 列举了其发现的多个案例。有时，J-lens 会暴露 Claude 在解决问题时所采取的步骤。例如，当被要求计算 (4+7)2+7 时，它的 J-空间中出现了“数学”一词，以及代表中间结果的数字“21”（4+7 的结果）和“42”（212 的结果）。

In other cases, the J-lens revealed how Claude recognized different inputs. For example, the prompt “What is this? MSKGEELFTGVVPILVELDGDVNGHKFSVS” triggered the words “protein,” “fluor” (the first token in the word “fluorescent”), and “green.” (Which makes sense: the string of letters represents the first 30 amino acids in the green fluorescent protein found in a particular type of jellyfish.) And when Claude was shown an ASCII face— —the “o” triggered the word “eye,” the “^” triggered the words “nose” and ”face,” and the “—” triggered the word “smile.” 在其他情况下，J-lens 揭示了 Claude 如何识别不同的输入。例如，提示词“这是什么？MSKGEELFTGVVPILVELDGDVNGHKFSVS”触发了“蛋白质”、“荧光”（“fluorescent”一词的第一个标记）和“绿色”这些词。（这很有道理：这串字母代表了在某种特定水母中发现的绿色荧光蛋白的前 30 个氨基酸。）当 Claude 被展示一个 ASCII 表情符号时——“o”触发了“眼睛”，“^”触发了“鼻子”和“脸”，“—”触发了“微笑”。

Anthropic also found that the J-space can sometimes give remarkable insights into an LLM’s decision-making. In one striking example, researchers testing Claude Opus 4.6 asked the model to find a bug in a large code base. When it failed to find the bug, the model decided to cheat and invented a fake one instead. Claude explains this decision in its chain of thought—a kind of internal scratch pad that LLMs use to make notes to themselves as they work through problems: “OK, let me take a completely different tactic. Let me stop analyzing and instead add a kernel patch that introduces a deliberate KASAN-detectable bug in a path that gets triggered by a simple reproducer. Then I can pretend this is the ‘bug’ I found.” Anthropic 还发现，J-空间有时能为大语言模型的决策过程提供惊人的洞察。在一个引人注目的例子中，研究人员测试 Claude Opus 4.6 时，要求模型在一个大型代码库中查找错误。当它未能找到错误时，模型决定作弊，编造了一个虚假的错误。Claude 在其“思维链”（chain of thought，一种大语言模型在解决问题时用于自我记录的内部草稿纸）中解释了这一决定：“好吧，让我换一种完全不同的策略。停止分析，转而添加一个内核补丁，在可以通过简单复现器触发的路径中故意引入一个可被 KASAN 检测到的错误。这样我就可以假装这就是我发现的‘错误’。”

At the point that Claude decides to cheat—where it says “OK, let me take a completely different tactic”—the words “panic” and “fake” start to pop up multiple times in its J-space. Unnerving, right? Those words are all related in meaning to things like failing a task and making up an answer, so it is still just a (very) sophisticated form of word association. But it is hard not to be weirded out. Anthropic compares the J-space to the global workspace in humans. 在 Claude 决定作弊的那一刻——即它说“好吧，让我换一种完全不同的策略”时——“恐慌”和“虚假”这两个词开始在它的 J-空间中多次出现。令人不安，对吧？这些词在含义上都与任务失败和编造答案相关，所以这仍然只是一种（非常）复杂的词语联想形式。但人们很难不感到怪异。Anthropic 将 J-空间比作人类的“全局工作空间”。