How a Frontier Model Gets Built, Read from the Kimi K3 Report
How a Frontier Model Gets Built, Read from the Kimi K3 Report
如何构建前沿模型：解读 Kimi K3 技术报告
Large Language Models: How a Frontier Model Gets Built, Read from the Kimi K3 Report. An open, 2.8-trillion-parameter model shipped with 47 pages of its own recipe. Reading it tells you what building a frontier model now involves, and how little of it is the model. 大型语言模型：如何构建前沿模型——解读 Kimi K3 技术报告。一个拥有 2.8 万亿参数的开源模型，随附了一份 47 页的“配方”。阅读这份报告，你就能了解构建前沿模型现在涉及哪些工作，以及模型本身在其中所占的比重有多小。
Comparison of standard transformer attention and Kimi Delta Attention. Full attention retains a KV cache that grows with sequence length, while KDA compresses information into a fixed-size state using a forget gate, allowing linear scaling. 标准 Transformer 注意力机制与 Kimi Delta 注意力机制的对比。全注意力机制保留随序列长度增长的 KV 缓存，而 KDA（Kimi Delta Attention）通过遗忘门将信息压缩为固定大小的状态，从而实现线性扩展。
You buy the model and its system card. The decisions that made it good stay in-house: how it was trained, what its reinforcement learning ran against, how they got it cheap enough to serve. That’s the normal deal with a frontier lab, and most outside writing about these systems has to work from the outputs alone. Kimi K3 is the exception worth reading. It’s open, it sits close to the best closed models on most benchmarks, and Moonshot published a forty-seven technical report that walks through the parts other labs keep private. I read it over an afternoon. What stayed with me is how small a share of the work is the model itself. 你买到的是模型及其系统卡（System Card）。但让模型表现出色的决策往往是内部机密：它是如何训练的、强化学习的对抗环境是什么、如何降低推理成本以实现商业化。这在顶尖实验室中是常态，大多数关于这些系统的外部研究只能基于输出结果进行推测。Kimi K3 是一个值得一读的例外。它是开源的，在大多数基准测试中表现接近顶尖的闭源模型，且月之暗面（Moonshot）发布了一份 47 页的技术报告，详细介绍了其他实验室通常保密的部分。我花了一个下午读完它，最让我印象深刻的是：模型本身在整个工作量中所占的比例竟如此之小。
One caveat up front. Everything specific below is Moonshot’s. I assume the closed labs do their own versions of the same categories of work, but I’m inferring that from the outside, so read the wider “this is what the frontier looks like” claims as a guess. 首先说明一点：以下所有具体内容均来自月之暗面。我推测闭源实验室也在进行类似的工作，但这只是我的外部推断，因此请将文中关于“前沿技术现状”的更广泛论述视为一种猜测。
The architecture is a stack of small changes
架构是一系列微小改动的堆叠
The architecture is the obvious place to start, and it holds the fewest surprises. K3 is a 2.8-trillion-parameter mixture-of-experts model, and what lifts it over the last Kimi is three fairly ordinary engineering changes stacked together. The attention keeps a fixed-size running state instead of a cache that grows with the input, which is what makes a million-token context affordable; most layers use that cheap version, and every fourth does full attention. 架构是显而易见的切入点，也是最不出人意料的部分。K3 是一个 2.8 万亿参数的混合专家模型（MoE），它之所以能超越上一代 Kimi，是因为堆叠了三个相当普通的工程改动。其注意力机制保留了一个固定大小的运行状态，而不是随输入增长的缓存，这使得百万 token 的上下文变得经济可行；大多数层使用这种低成本版本，每四层进行一次全注意力计算。
Positions aren’t encoded explicitly, so the recurrence has to carry them, which lets the model stretch to a million tokens without the usual rescaling hacks. Each layer can also look back at every layer beneath it, not only the one directly below, so early signal doesn’t wash out on the way up. And each token is routed to 16 of 896 experts, sparser than before, which takes real care to keep stable. Moonshot reports a 2.5× gain in scaling efficiency over Kimi K2, roughly the same quality for under half the training compute. It credits the architecture together with refined data and training recipes, without apportioning the gain between them, so don’t read the 2.5× as three architecture changes multiplying out. Each piece is an ordinary, well-tested idea rather than a single breakthrough. 位置信息不再被显式编码，而是由循环机制携带，这使得模型无需常规的重缩放技巧就能扩展到百万 token。每一层还可以回顾其下方的所有层，而不仅仅是紧邻的下一层，从而防止早期信号在向上传递过程中被稀释。此外，每个 token 被路由到 896 个专家中的 16 个，比以往更稀疏，这需要极高的稳定性控制。月之暗面报告称，其扩展效率比 Kimi K2 提升了 2.5 倍，即以不到一半的训练算力达到了大致相同的质量。他们将这一提升归功于架构、精炼数据和训练配方的共同作用，并未具体拆分各部分的贡献，因此不要将 2.5 倍简单理解为三个架构改动的乘积。每一个部分都是经过充分验证的常规思路，而非单一的突破。
The three changes in plain English
用通俗语言解释这三个改动
Each of the three is easy to hold in your head once you drop the notation, and the plain version is where the intuition lives. Attention is a lookup table, and that’s why long context is expensive. A transformer handles each token by looking back over every earlier token and pulling a weighted blend of them, a soft lookup. To do that it keeps a small key-and-value record for every token so far, the KV cache. Think of a filing cabinet that never throws a card away: each new word files a card, then flicks through every card already in the drawer to decide what to attend to. Fine at a few thousand words. At a million, the drawer is enormous and every new word riffles the entire thing, so the cost climbs with the square of the length. 一旦抛开复杂的符号，这三个改动都很容易理解，通俗版本更能体现其直觉。注意力机制本质上是一个查找表，这就是长上下文昂贵的原因。Transformer 处理每个 token 的方式是回顾之前的所有 token，并提取它们的加权混合，这是一种“软查找”。为了做到这一点，它必须为迄今为止的每个 token 保留一个键值记录（KV 缓存）。想象一个永远不会丢弃卡片的档案柜：每出现一个新词，就存入一张卡片，然后翻阅抽屉里所有的卡片来决定关注什么。在几千字的情况下这没问题，但在百万字规模下，抽屉变得巨大，每出现一个新词都要翻阅整个档案，因此成本随长度的平方增长。
K3’s main attention swaps the cabinet for a single running summary, a fixed-size notepad it writes over as it goes. The notepad never grows, and that fixed size is what makes a million-token context affordable. What you give up is that a summary can’t keep everything, so it has to forget, and K3 hands it a per-feature dial for how fast old detail fades. A summary also can’t recall an exact earlier token on demand, so K3 keeps one true cabinet layer for every three notepad layers and buys the precise recall back where it counts. K3 的主要注意力机制将档案柜换成了一个单一的运行摘要，即一个随写随改的固定大小记事本。记事本永远不会变大，这种固定大小正是百万 token 上下文变得经济的原因。你放弃的是摘要无法保留所有细节，因此它必须“遗忘”，而 K3 为每个特征提供了一个调节旋钮，控制旧细节的衰减速度。此外，摘要无法按需精确回溯之前的某个 token，因此 K3 在每三个记事本层中保留一个真正的档案柜层，从而在关键位置找回了精确回溯能力。
Figure 1: Two ways to mix over tokens. Full attention stores a key/value entry per token and compares each new token against all of them (cost ∝ n²); Kimi Delta Attention folds tokens into one fixed-size state through a forget gate α (cost ∝ n). K3 stacks three linear layers per global layer. 📖 Source: image by author. 图 1：混合 token 的两种方式。全注意力机制为每个 token 存储一个键/值条目，并将每个新 token 与所有旧 token 进行比较（成本 ∝ n²）；Kimi Delta Attention 通过遗忘门 α 将 token 折叠成一个固定大小的状态（成本 ∝ n）。K3 在每个全局层中堆叠了三个线性层。📖 来源：作者绘图。
The fixed-state layers already track order as they update, which lets K3 drop explicit positional encodings entirely. Most models bolt position onto attention with something like RoPE, and stretching the context window later means rescaling those frequencies or interpolating them, which is fiddly and lossy. K3’s recurrence carries position for free, so the same weights run at 8K during early training and at 1M after the long-context stage with no positional surgery in between. 固定状态层在更新时会自动跟踪顺序，这使得 K3 可以完全放弃显式的位置编码。大多数模型通过 RoPE 等方式将位置信息“钉”在注意力机制上，而后续扩展上下文窗口意味着需要重缩放或插值这些频率，这既繁琐又有损精度。K3 的循环机制免费携带了位置信息，因此同一套权重在早期训练时运行在 8K 上下文，在长上下文阶段运行在 1M 上下文，中间无需进行任何位置相关的“手术”。
The same lookup trick, one level up. Stack a lot of layers and each one normally adds its output to a shared running total that flows up the network. Picture that total as a notepad passed up a line of people, each scribbling a line. By the top the first few lines are buried, and the network burns capacity just keeping early information alive. Attention Residuals let a layer reach back and read earlier layers’ outputs directly, weighting them with a softmax, the same soft-lookup machinery as attention but pointed across depth instead of across the sequence. A layer pulls a blend from source instead of playing telephone up the stack. The cost is memory rather than new machinery: every layer’s output has to stay live for the ones above it, so K3 attends over a handful of block summaries instead of all of its ninety-odd layers to keep that bill down. 同样的查找技巧，提升了一个维度。堆叠大量层时，每一层通常将其输出添加到一个向上传递的共享运行总和中。想象这个总和是一本在人群中传递的记事本，每个人都在上面写一行字。传到顶端时，最初的几行字已经被掩盖了，网络为了保持早期信息而消耗了大量容量。注意力残差（Attention Residuals）允许某一层直接回溯并读取早期层的输出，通过 Softmax 进行加权——这与注意力机制中的软查找机制相同，只是方向从序列维度转向了深度维度。每一层直接从源头提取混合信息，而不是像玩“传声筒”游戏那样逐层传递。其代价是内存而非新机制：每一层的输出必须为上层保持活跃，因此 K3 只关注少数几个块摘要，而不是全部九十多个层，从而控制了内存开销。