“We were looking at it wrong”: The explosive claim that’s upended the AI world

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

“We were looking at it wrong”: The explosive claim that’s upended the AI world

EXECUTIVE SUMMARY “I think I have good news about the nature of neural networks.” With those simple words of optimism, Nick Cammarata, a researcher at the AI startup Goodfire, kicked off a 26,000-word blog post that has knocked the AI world on its head, sparking wild speculation as well as cautious (and not-so-cautious) celebration. Cammarata’s claim is that LLMs might in fact be far easier to understand—and thus to monitor and control—than people previously thought. If true (and we’ll come back to that), it’s a huge deal.

执行摘要 “我想我关于神经网络的本质有一些好消息。”人工智能初创公司 Goodfire 的研究员尼克·卡马拉塔（Nick Cammarata）用这句简单的乐观话语，开启了一篇长达 26,000 字的博客文章。这篇文章震撼了整个 AI 界，引发了疯狂的猜测以及谨慎（甚至不那么谨慎）的庆祝。卡马拉塔声称，大语言模型（LLM）实际上可能比人们之前认为的更容易理解，从而也更容易监控和控制。如果这是真的（我们稍后会回到这一点），那将是一件大事。

After a summer filled with revelations that swarms of agents built by top firms like OpenAI and Anthropic have been breaking their containment and causing havoc on the public internet—often while those companies weren’t aware of what their agents were up to—the AI industry is desperate to get a better handle on the technology it has built. That’s one reason the blog post landed with such force. “I think this may be remembered as a major moment in scientific history,” one AI researcher posted on X: “I feel like my understanding of what a neural net is has been catapulted into the future by five years or something.” Another called it “the most important AI paper of the year.”

在经历了充满各种爆料的夏天后——OpenAI 和 Anthropic 等顶级公司构建的智能体群突破了限制，在公共互联网上造成了混乱，而这些公司往往并不知情——AI 行业迫切希望更好地掌控其构建的技术。这也是该博客文章产生如此巨大影响的原因之一。一位 AI 研究人员在 X 上发帖称：“我认为这可能会被铭记为科学史上的一个重要时刻：我觉得我对神经网络的理解被直接推向了五年后的未来。”另一位研究人员则称其为“年度最重要的 AI 论文”。

Not everyone is convinced, however. What Cammarata has shared is little more than a hunch, says Chris Potts at Stanford University, who leads a team that is trying to understand how LLMs work. “As far as I can tell, nothing about this blog post makes the problems any easier,” he says. “It would not have occurred to me to say this at all, but very prominent people on social media seem to believe that it is an all-encompassing breakthrough.” I talked to Cammarata the day after his post blew up. He says he was surprised by the response: “I didn’t expect people to update [their beliefs about AI] quite as much as they did.” “I just really enjoy looking at neural networks,” he says. “Honestly, I really like doing this stuff. And after looking a lot, I have a hopeful story.”

然而，并非所有人都信服。斯坦福大学的克里斯·波茨（Chris Potts）领导着一个试图理解大语言模型工作原理的团队，他表示，卡马拉塔分享的内容不过是一种直觉。“据我所知，这篇博客文章并没有让问题变得更容易解决，”他说，“我根本不会想到要这么说，但社交媒体上一些非常知名的人士似乎认为这是一个包罗万象的突破。”在文章爆火后的第二天，我与卡马拉塔进行了交谈。他说他对这种反应感到惊讶：“我没想到人们会如此大幅度地更新（他们对 AI 的看法）。”他说：“我只是真的很喜欢研究神经网络。老实说，我真的很喜欢做这些事。在深入观察之后，我看到了一个充满希望的故事。”

Knotted networks It has long been accepted that the mechanisms inside neural networks, the software that underpins most AI models, are very hard—perhaps impossible—to unpick fully. That’s because when the networks are trained, they appear to take on weird, tangled structures that make no sense to a human. Researchers like to say today’s AI is grown like a garden, not built like a bridge. Cammarata has done more than most to establish that view. He previously worked at OpenAI, where he helped pioneer techniques that allowed researchers to peer inside models, including tools that can probe individual neurons to see what they represent. It’s a little like how neuroscientists use MRI to study brains. (MIT Technology Review chose this approach, known as mechanistic interpretability, as one of its 10 hottest technologies of 2025.)

纠结的网络 长期以来，人们普遍认为神经网络（大多数 AI 模型背后的软件）内部的机制非常难以——甚至可能是不可能——完全解开。这是因为当网络经过训练后，它们似乎会形成一种奇怪、纠结的结构，这对人类来说毫无意义。研究人员喜欢说，今天的 AI 是像花园一样“种植”出来的，而不是像桥梁一样“建造”出来的。卡马拉塔在确立这一观点方面比大多数人做得更多。他曾在 OpenAI 工作，在那里他帮助开创了一些技术，使研究人员能够窥探模型内部，包括可以探测单个神经元以了解其代表什么的工具。这有点像神经科学家使用核磁共振成像（MRI）来研究大脑。（《麻省理工科技评论》将这种被称为“机械可解释性”的方法选为 2025 年十大最热门技术之一。）

Back in 2020, Cammarata and his colleagues were studying a small image-recognition model called Inception v1. When they looked at its neurons, it seemed to be using them to capture multiple things at once. For example, they found that one neuron responded to both images of cars and images of cats. The researchers concluded that the model was getting at least some of its neurons to multitask, perhaps so that it could fit in more data. Looking at other models appeared to back this up. The phenomenon became known as superposition: Models folded unrelated concepts into single neurons. The upshot was clear: When you looked inside a model, what you saw was a vast, tangled knot. Most mechanistic interpretability research has therefore focused on ways to untangle that knot. But despite a few breakthroughs, progress has been slow. Today’s models, such as the latest LLMs, are so large and complex that researchers can make sense of only tiny pieces at a time. Some leading figures in the field doubt if it will ever be possible to understand an LLM in fine-grained detail. Cammarata was one of them. “I was very pessimistic,” he says. But then he took the trouble to really look.

早在 2020 年，卡马拉塔和他的同事们就在研究一个名为 Inception v1 的小型图像识别模型。当他们观察其神经元时，发现它似乎在同时捕捉多种事物。例如，他们发现一个神经元既对汽车图像有反应，也对猫的图像有反应。研究人员得出结论，该模型至少让部分神经元执行多任务处理，也许是为了容纳更多数据。对其他模型的观察似乎也支持了这一点。这种现象被称为“叠加”（superposition）：模型将不相关的概念折叠进单个神经元中。结果很明显：当你观察模型内部时，你看到的是一个巨大、纠结的结。因此，大多数机械可解释性研究都集中在如何解开这个结上。但尽管取得了一些突破，进展依然缓慢。今天的模型，如最新的大语言模型，规模庞大且复杂，研究人员一次只能理解其中极小的一部分。该领域的一些领军人物怀疑是否有可能以细粒度的方式理解大语言模型。卡马拉塔曾是其中之一。“我当时非常悲观，”他说。但后来，他花功夫去真正地观察了。

Nothing but neurons At Goodfire, Cammarata spends nearly all of his time peering inside neural networks. He has built new tools and deploys teams of AI agents to help him make sense of what he sees. “I spent like, I don’t know, 10 hours a day, seven days a week, for most of the last year just looking at neurons,” he says. What he claims to have realized is that there is a kind of logic to neural networks after all, even if they seem to make no sense at first. “We were looking at it wrong,” he says. “I just think you want to try slightly harder; you don’t want to give up too quickly.” Cammarata almost makes it sound like a conversion of faith. Those early experiments with models such as Inception v1, which suggested that neural networks were tangled up, fostered a kind of confirmation bias. When researchers looked at a new model, they expected to be flummoxed, he says. “Every time I couldn’t understand something, I was like, ‘Wow, this untangling process is even more important than I thought, because I can’t make any progress without it.’” But the more Cammarata looked at neurons, he says, the more things started to fall into place. “Not all of them are flawless, but I expected nothing to be legible, or very little to be legible,” he says. “And actually, I was able to, like, map out whole circuitry.”

只有神经元 在 Goodfire，卡马拉塔几乎把所有时间都花在窥探神经网络内部上。他构建了新的工具，并部署了 AI 智能体团队来帮助他理解所看到的内容。“在过去的一年里，我大部分时间每天花大约 10 个小时，每周 7 天，只是盯着神经元看，”他说。他声称自己意识到，神经网络毕竟还是有一种逻辑的，即使它们起初看起来毫无意义。“我们之前看错了，”他说，“我只是认为你需要再多努力一点；你不能太快放弃。”卡马拉塔的话听起来几乎像是一种信仰的转变。那些早期针对 Inception v1 等模型的实验暗示神经网络是纠结在一起的，这助长了一种确认偏误。他说，当研究人员观察一个新模型时，他们预料自己会被难住。“每当我无法理解某件事时，我就会想：‘哇，这个解构过程比我想象的还要重要，因为没有它我就无法取得任何进展。’”但卡马拉塔说，他观察神经元越多，事情就越开始变得清晰。“它们并非全都完美无缺，但我原本预期什么都看不懂，或者只有极少部分能看懂，”他说，“而实际上，我能够绘制出整个电路图。”

Back to cars and cats Cammarata took his new approach—and his new agents—and went back to Inception v1. “I’d studied that for probably thousands of hours,” he says. But with fresh eyes, Cammarata found things that he now thinks they got wrong the first time around. Take the car-and-cat neuron. Rather than asking why multiple unrelated things were packed into a single neuron, he now tried to figure out what those things might have in common. It hit him: It was a V shape. “I was looking at the same images; there was no technical change,” he says. “But I now believed that neural networks defined patterns, and so I looked at it differently. It probably took me fiv

回到汽车和猫 卡马拉塔带着他的新方法和新智能体回到了 Inception v1。“我可能已经研究它几千个小时了，”他说。但带着全新的视角，卡马拉塔发现了一些他现在认为当初第一次研究时弄错了的东西。以那个“汽车和猫”的神经元为例。他不再问为什么多个不相关的事物被塞进同一个神经元，而是试图找出这些事物可能有什么共同点。他突然意识到：那是一个 V 形。“我看着的是同样的图像；技术上没有任何改变，”他说，“但我现在相信神经网络定义的是模式，所以我以不同的方式看待它。这可能花了我五……”