What Anthropic’s latest AI discovery does—and doesn’t—show

Anthropic 最新的人工智能发现揭示了什么（以及没揭示什么）

Anthropic—currently the world’s most valuable AI company, with a nearly $1 trillion valuation—has a reputation for publishing strange and heady research. It’s looking into whether AI models can feel pain, for example, and will sometimes cut off chatbot conversations if it suspects users are “abusing” the model. Anthropic 目前是全球估值最高的人工智能公司，估值接近 1 万亿美元，以发布一些奇特且深奥的研究而闻名。例如，它正在研究人工智能模型是否能感受到痛苦，有时如果怀疑用户在“滥用”模型，它还会切断与聊天机器人的对话。

One niche that Anthropic spends more time and money on than other AI companies is called mechanistic interpretability, which means looking inside the complex math of an AI model to learn why it comes up with one particular output and not another. It’s complicated stuff; there are millions of data points that might contribute to any result, and wading through them can look more like word salad than anything useful. Anthropic 在一个细分领域投入的时间和资金比其他 AI 公司都要多，这个领域被称为“机械可解释性”（mechanistic interpretability）。这意味着深入研究 AI 模型复杂的数学原理，以了解它为什么会得出特定的输出而不是其他结果。这是一项复杂的工作；有数百万个数据点可能对任何结果产生影响，梳理这些数据看起来更像是毫无意义的“词语沙拉”，而非有用的信息。

It’s also controversial. Describing AI models with terms borrowed from psychology and neuroscience can make their behavior seem more sophisticated than we might otherwise judge it to be. That’s why, when Anthropic announced last week that it had found a new window into its models’ “internal thoughts” as they reason through answers, there was one colleague I had to talk to. 这也颇具争议。用借自心理学和神经科学的术语来描述 AI 模型，可能会让它们的行为看起来比我们实际判断的要复杂得多。正因如此，当 Anthropic 上周宣布它发现了一个观察模型在推理答案时“内在思想”的新窗口时，我必须找一位同事聊聊。

Senior editor Will Douglas Heaven, aside from having a PhD in computer science, has spent a lot of time digging into what we can say about how AI models work. I spoke with him about what we should take from Anthropic’s new (and predictably quirky) research. 资深编辑 Will Douglas Heaven 不仅拥有计算机科学博士学位，还花费了大量时间深入研究 AI 模型的工作原理。我与他探讨了我们应该如何看待 Anthropic 这项新（且一如既往地古怪）的研究。

What did Anthropic learn here, exactly? Anthropic has been trying to understand how large language models (LLMs) work for a few years now. Anthropic isn’t the only one looking at this, but I think the company has made it part of its core mission more than most. Anthropic’s CEO, Dario Amodei, has said we won’t be able to control LLMs fully unless we learn more about how they work. So this new research is very much in that context. It goes deeper into the weird mechanisms inside LLMs than ever before. Anthropic 到底发现了什么？几年来，Anthropic 一直试图理解大语言模型（LLM）的工作原理。并非只有 Anthropic 在研究这个问题，但我认为该公司比大多数同行更将其视为核心使命。Anthropic 的首席执行官 Dario Amodei 曾表示，除非我们更深入地了解 LLM 的工作方式，否则无法完全控制它们。因此，这项新研究正是在这一背景下进行的。它比以往任何时候都更深入地探索了 LLM 内部那些奇特的机制。

What Anthropic learned was that LLMs have a space inside them—which Anthropic calls the J-space—filled with words that don’t appear in their output but that seem to influence the way they puzzle through problems. All this was hidden until Anthropic developed a new technique to probe its model Claude, so it’s a genuine discovery. Anthropic 的发现是，LLM 内部存在一个空间——Anthropic 将其称为“J-空间”（J-space）——里面充满了不会出现在输出结果中，但似乎会影响模型解决问题方式的词汇。在 Anthropic 开发出探测其模型 Claude 的新技术之前，这一切都是隐藏的，因此这是一个真正的发现。

Sometimes these words keep track of where the LLM has got to in a particular task, sometimes they look more like flashes of recognition (for example, “protein” might pop up when you give an LLM only the letters of a protein sequence), and sometimes they represent a kind of internal commentary on the model’s decision-making. In my favorite example, Claude decided to cheat on a coding test when the word “panic” appeared. Anthropic also found that LLMs are able to describe and manipulate the words in this space. So somehow they seem to be making use of it. 有时，这些词汇会记录 LLM 在特定任务中的进度；有时它们看起来更像是认知的闪现（例如，当你只给 LLM 提供蛋白质序列的字母时，“蛋白质”一词可能会弹出）；有时它们代表了模型对自身决策的一种内在评论。我最喜欢的一个例子是，当“恐慌”一词出现时，Claude 决定在编程测试中作弊。Anthropic 还发现，LLM 能够描述并操纵这个空间里的词汇。因此，它们似乎在某种程度上利用了这个空间。

Let’s step back for a second. I don’t think of large language models as simple, but they’re also not magic. There’s a bunch of math that learns relationships between words, right? So why is it so hard to “peer” into an LLM to know what’s going on? 让我们退一步思考。我不认为大语言模型很简单，但它们也不是魔法。它们是一堆学习词汇之间关系的数学模型，对吧？那么，为什么“窥探”LLM 以了解其内部运作机制如此困难呢？

Yeah, they’re not magic! I think the fact we don’t fully understand them plays into the mythmaking. And it’s worth noting that the whole narrative that Anthropic is leaning into here—that they’ve built this really mysterious technology, but don’t worry, because they’re also the ones to figure it out—very much fits with the company’s vibe. So yes: LLMs are just math. And yet it’s vastly complex math. Not only are today’s LLMs made out of hundreds of billions of numbers, but running them triggers a cascade of millions and millions of calculations. 是的，它们不是魔法！我认为我们对它们缺乏完全理解，这助长了神话色彩。值得注意的是，Anthropic 在这里所强调的叙事——即他们构建了这种极其神秘的技术，但别担心，因为他们也是唯一能破解它的人——非常符合该公司的风格。所以，是的：LLM 只是数学。但这是极其复杂的数学。如今的 LLM 不仅由数千亿个数字组成，而且运行它们还会触发数以百万计的级联计算。

I wrote last year that if you printed out even a medium-size LLM on pieces of paper, it would cover a city the size of San Francisco. It’s impossible to make sense of any of that math without specialist tools that highlight specific parts of an LLM at specific times. You need to know where to look and how to look. And building those tools requires understanding something of that complex math in the first place. 我去年写过，如果你把一个中等规模的 LLM 打印在纸上，它能覆盖整个旧金山市。如果没有专门的工具在特定时间突出显示 LLM 的特定部分，就不可能理解其中的任何数学逻辑。你需要知道看哪里以及如何看。而构建这些工具的前提，首先需要理解那些复杂的数学原理。

You’ve written elsewhere about this concept of studying LLMs the way one might study an organism’s brain. Is it fair to use “brain-like” terms when talking about how an LLM works? 你在其他地方写过关于像研究生物大脑一样研究 LLM 的概念。在谈论 LLM 的工作原理时，使用“类脑”术语公平吗？

I don’t love using those kinds of terms. LLMs are not brains. Talking like this is misleading because it can suggest that LLMs are capable of more human-like things than they are or that we can make assumptions about how they might behave that we shouldn’t. The whole anthropomorphization thing is also tied up with a bunch of strong ideological positions about what this technology is and what it’s going to be. But at the same time, we lack a good alternative vocabulary for talking about what these models are doing. I can understand why people reach for words like “think” and “understand” and “brain-like”—they’re convenient shorthand. 我不喜欢使用这类术语。LLM 不是大脑。这样说具有误导性，因为它可能暗示 LLM 具备比实际能力更像人类的特质，或者让我们对它们的行为做出不该有的假设。整个拟人化的问题也与关于这项技术是什么及其未来走向的一系列强烈意识形态立场纠缠在一起。但与此同时，我们缺乏一套好的替代词汇来描述这些模型在做什么。我能理解为什么人们会使用“思考”、“理解”和“类脑”这样的词——它们是方便的速记。

Anthropic compares this new space it found inside LLMs to the space that some neuroscientists think our brains use to keep track of conscious thoughts. I asked the company how seriously we should take that comparison and it said in a statement: “Drawing these analogies was helpful to us in designing our experiments, as they allowed us to make many non-obvious experimental predictions about the J-space that turned out to be true. At the same time, it’s important to note that there are some important differences between the J-space (and language models in general) and the human brain, so we don’t mean to claim there’s a perfect correspondence.” Anthropic 将其在 LLM 内部发现的这个新空间，比作一些神经科学家认为我们大脑用来追踪意识思维的空间。我询问该公司我们应该多认真地对待这种比较，它在一份声明中表示：“做出这些类比对我们设计实验很有帮助，因为它们使我们能够对 J-空间做出许多非显而易见的实验预测，结果证明这些预测是正确的。同时，必须指出的是，J-空间（以及一般的语言模型）与人脑之间存在一些重要差异，因此我们并不打算声称两者之间存在完美的对应关系。”

What’s a problem in AI that this new concept of the J-space might be used to solve? Anthropic has said that monitoring the J-space could be a way to catch models doing something they shouldn’t. Because words pop up in this space that don’t… J-空间这一新概念可能被用来解决 AI 领域的什么问题？Anthropic 表示，监控 J-空间可能成为捕捉模型违规行为的一种方式。因为在这个空间中弹出的词汇并不……