These startups are chasing the next big thing in LLMs
These startups are chasing the next big thing in LLMs
这些初创公司正在追逐大语言模型（LLM）的下一个风口
EXECUTIVE SUMMARY MIT Technology Review’s What’s Next series looks across industries, trends, and technologies to give you a first look at the future. You can read the rest of them here. 执行摘要 《麻省理工科技评论》的“未来趋势”（What’s Next）系列深入探讨各行各业、趋势和技术，带您抢先一睹未来。您可以在此处阅读该系列的其他文章。
Way back in the summer of 2017, AI researchers at Google put out a paper called “Attention Is All You Need,” in which they described a new type of neural network called a transformer. It proved to be very good at processing long sequences of data, especially text. Nine years on, transformers are the engines inside every major large language model on the market. 早在 2017 年夏天，谷歌的 AI 研究人员发表了一篇名为《注意力即一切》（Attention Is All You Need）的论文，其中描述了一种名为 Transformer 的新型神经网络。事实证明，它在处理长序列数据（尤其是文本）方面表现极其出色。九年过去了，Transformer 已成为市场上每一个主流大语言模型背后的引擎。
“The entire AI industry is built on transformers,” says Justin Dangel, cofounder and CEO of the AI startup Subquadratic. “They are one of the most important innovations in the history of computer science, and they’ve changed the world.” “整个 AI 行业都是建立在 Transformer 之上的，”AI 初创公司 Subquadratic 的联合创始人兼首席执行官贾斯汀·丹格尔（Justin Dangel）表示，“它们是计算机科学史上最重要的创新之一，并且已经改变了世界。”
But transformers are starting to show their age. Many of the recent advances in LLMs, such as the development of so-called reasoning models and their ability to handle large amounts of input at once, are not neat extensions of that core technology but workarounds that patch over some of its fundamental flaws. 但 Transformer 开始显露出疲态。近期大语言模型的许多进展，例如所谓“推理模型”的开发及其同时处理大量输入的能力，并非该核心技术的完美延伸，而是为了修补其根本缺陷而采取的变通方案。
A growing number of scientists and engineers are now asking what’s coming next. LLMs are not going anywhere, but the way they get built is up for grabs. (MIT Technology Review dubbed this future generation of models LLMs+ in this year’s list of the 10 things that matter in AI.) 越来越多的科学家和工程师开始思考：下一个突破口在哪里？大语言模型不会消失，但它们的构建方式正面临重塑。（《麻省理工科技评论》在今年的“全球十大突破性技术”榜单中，将这一代未来的模型称为 LLMs+。）
Enter a wave of startups hoping to push the boundaries of this boomtown technology. Some will no doubt fail—but they have everything to play for and far less to lose than the companies at the front of the pack today. 于是，一批初创公司涌现出来，希望突破这项热门技术的边界。毫无疑问，其中一些会失败，但它们拥有无限的机遇，且相比目前处于行业领先地位的公司，它们失去的更少。
Strength in numbers 人多力量大
But first, the problem. The key strength of transformers lies in a mechanism called dense attention, which encodes the meaning of a block of text in a series of numbers. The process involves comparing every word (or part of a word, known as a token) in that text with every other word via a form of multiplication. 首先，让我们看看问题所在。Transformer 的核心优势在于一种名为“密集注意力”（dense attention）的机制，它将一段文本的含义编码为一系列数字。该过程涉及通过某种乘法形式，将文本中的每个单词（或单词的一部分，即 Token）与所有其他单词进行比较。
Dense attention can capture the meaning of text with remarkable accuracy. But as the length of that text grows, the number of computations needed to process it adds up fast. A document 10,000 words long might require a transformer to perform 50 million multiplications. That’s the main reason LLMs suck up so much power. 密集注意力机制能够以极高的准确度捕捉文本含义。但随着文本长度的增加，处理它所需的计算量会迅速累积。一篇 1 万字的文章可能需要 Transformer 执行 5000 万次乘法运算。这就是大语言模型消耗如此多电力的主要原因。
The costs are huge. OpenAI is set to spend $50 billion on computing this year, according to the company’s president, Greg Brockman. And the International Energy Agency predicts that the total amount of electricity consumed by data centers will double by 2030. 成本是巨大的。据 OpenAI 总裁格雷格·布罗克曼（Greg Brockman）称，该公司今年在计算方面的支出预计将达到 500 亿美元。国际能源署预测，到 2030 年，数据中心的总耗电量将翻一番。
What’s more, transformers struggle with what many of the latest models are designed to do. Because of the way they process text word by word, transformers are not great at keeping track of a lot of information at once (in other words, what’s known as their context window cannot get too large). 此外，Transformer 在处理许多最新模型所需的功能时显得力不从心。由于它们逐字处理文本的方式，Transformer 不擅长同时跟踪大量信息（换句话说，即它们的“上下文窗口”无法无限扩大）。
And yet if LLMs are to carry out harder tasks, they will need to take in larger amounts of data: a whole library of documents, an entire code base, or in the case of agents, output from other LLMs. As for reasoning models, they work by writing notes to themselves (in a kind of scratch pad known as a chain of thought) and then reading them back, which again adds to the amount of data to stay on top of. 然而，如果大语言模型要执行更艰巨的任务，它们就需要摄入更多数据：整个图书馆的文档、完整的代码库，或者在智能体（Agent）的情况下，摄入其他大语言模型的输出。至于推理模型，它们通过给自己写笔记（在一种被称为“思维链”的草稿纸上）并回读这些笔记来工作，这再次增加了需要处理的数据量。
As LLMs get bigger and better, transformers have become a bottleneck. The technology’s key strength is now a limitation. Here are four new ideas for how to solve the transformer problem—innovations that could change LLMs for good, making them faster, far more efficient, and (maybe) even smarter. 随着大语言模型变得越来越大、越来越强，Transformer 已成为瓶颈。该技术的关键优势现在反而成了限制。以下是解决 Transformer 问题的四种新思路——这些创新可能会彻底改变大语言模型，使其运行速度更快、效率更高，甚至（可能）更聪明。
01: Rethinking attention 01：重新思考注意力机制
An obvious way to make LLMs faster and cheaper is to tackle the problem head on and change the way attention works. Swapping out dense attention for a mechanism called sparse attention, which runs calculations on only some pairings of words in a block of text instead of all of them, can radically reduce the amount of computation LLMs need to do. 让大语言模型更快、更便宜的一个显而易见的方法是直面问题，改变注意力机制的工作方式。将密集注意力替换为一种名为“稀疏注意力”（sparse attention）的机制——它仅对文本块中部分单词对进行计算，而非全部——可以从根本上减少大语言模型所需的计算量。
Researchers have come up with plenty of sparse attention mechanisms over the years. The problem is that none of them were as good as dense attention at capturing meaning. That might have changed. Subquadratic, a startup based in Miami, claims it has invented the first sparse attention mechanism that rivals top mainstream LLMs on a handful of tasks, including search and coding. 多年来，研究人员提出了许多稀疏注意力机制。问题在于，它们在捕捉语义方面都不如密集注意力。这种情况可能已经改变了。总部位于迈阿密的初创公司 Subquadratic 声称，他们发明了首个在搜索和编码等少数任务上能与顶级主流大语言模型相媲美的稀疏注意力机制。
It’s a huge claim (and some people in the industry remain skeptical). Subquadratic says its model, SubQ, works by figuring out on the fly—for each piece of text it is given—which words matter and which don’t. The company also claims that thousands have signed up to its waitlist and plans to make the model widely available soon. 这是一个大胆的声明（业内一些人仍持怀疑态度）。Subquadratic 表示，其模型 SubQ 的工作原理是针对给定的每一段文本，实时判断哪些单词重要，哪些不重要。该公司还声称已有数千人加入了其候补名单，并计划很快将该模型广泛发布。
Meanwhile, Manifest AI, a startup based in San Francisco, is coming at the problem from a different angle. Instead of changing how attention works, it is replacing it with something else. It has developed a mechanism it calls power retention, which stores only the most relevant information for a given task and ensures that the amount of data an LLM has to keep track of doesn’t blow up. 与此同时，总部位于旧金山的初创公司 Manifest AI 正从另一个角度切入。他们没有改变注意力机制，而是用其他东西取而代之。他们开发了一种名为“功率保留”（power retention）的机制，该机制仅存储给定任务中最相关的信息，并确保大语言模型需要跟踪的数据量不会激增。
Attention mechanisms force LLMs to keep track of everything in their context window. A sparse attention model (such as SubQ) throws out a lot of the individual words, but it still retains a rough picture of everything it has seen. In contrast, power retention works by providing the model with a rolling summary of its context window. As new information is added, less relevant information is dropped. 注意力机制迫使大语言模型跟踪上下文窗口中的所有内容。稀疏注意力模型（如 SubQ）会丢弃许多单个单词，但仍保留其所见内容的粗略概览。相比之下，“功率保留”的工作方式是为模型提供上下文窗口的滚动摘要。随着新信息的加入，不太相关的信息会被丢弃。
The basic principle of retention has been around for a decade. Manifest AI claims it has updated those techniques to build models that can stand up to transformer-based LLMs for the first time. The company says it is possible to adapt a transformer model into a power retention model with minimal retraining. To demonstrate this, it has turned an existing open-source coding LLM called StarCoder into a version that uses power retention, called PowerCoder. It has also released a model called Brumby, which it claims rivals some versions of Alibaba’s popular open-source model Qwen. Manifest AI wants its power retention… “保留”的基本原理已经存在十年了。Manifest AI 声称他们已经更新了这些技术，首次构建出能够与基于 Transformer 的大语言模型相抗衡的模型。该公司表示，只需极少的重新训练，就有可能将 Transformer 模型改造为“功率保留”模型。为了证明这一点，他们将现有的开源编码大语言模型 StarCoder 改造为使用“功率保留”的版本，命名为 PowerCoder。他们还发布了一个名为 Brumby 的模型，并声称其性能可与阿里巴巴流行的开源模型 Qwen 的某些版本相媲美。Manifest AI 希望其“功率保留”技术……