Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
Kimi Linear：一种具有表现力且高效的注意力架构 (2025)
Abstract: We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios — including short-context, long-context, and reinforcement learning (RL) scaling regimes.
摘要： 我们推出了 Kimi Linear，这是一种混合线性注意力架构。在包括短上下文、长上下文以及强化学习（RL）扩展在内的多种场景下，该架构首次在公平对比中超越了全注意力机制（Full Attention）。
At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory.
其核心是 Kimi Delta Attention (KDA)，这是一个具有表现力的线性注意力模块。它通过更细粒度的门控机制扩展了 Gated DeltaNet，从而能够更有效地利用有限的有限状态 RNN 内存。
Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule.
我们定制的分块算法通过一种特殊的对角加低秩（DPLR）转移矩阵变体，实现了极高的硬件效率。与通用的 DPLR 公式相比，该方法在大幅降低计算量的同时，与经典的 Delta 规则保持了更好的一致性。
We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA).
我们预训练了一个 Kimi Linear 模型，该模型拥有 30 亿激活参数和 480 亿总参数，基于 KDA 和多头潜在注意力（MLA）的逐层混合架构。
Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context.
实验表明，在相同的训练配方下，Kimi Linear 在所有评估任务中均以显著优势超越了全 MLA 架构，同时将 KV 缓存使用量降低了高达 75%，并在 100 万上下文长度下实现了高达 6 倍的解码吞吐量。
These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths.
这些结果证明，Kimi Linear 可以作为全注意力架构的直接替代方案，在包括更长输入和输出长度的任务中提供卓越的性能和效率。
To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.
为了支持后续研究，我们开源了 KDA 内核和 vLLM 实现，并发布了预训练及指令微调后的模型检查点。