Before Q, K, and V: Reconstructing the Transformer
Before Q, K, and V: Reconstructing the Transformer
在 Q、K 和 V 之前：重构 Transformer
Deep Learning Before Q, K, and V: Reconstructing the Transformer. Many Transformer explainers start with the finished architecture. We ask why it looks the way it does. 在 Q、K 和 V 之前的深度学习：重构 Transformer。许多关于 Transformer 的解释文章都从最终的架构开始讲起，而我们要探讨的是：它为什么会是现在这个样子？
Do we really need keys, queries, values, and dot product attention? “You need keys and queries for tokens to talk to each other,” says the popular Internet analogy. But why? There’s a lot of great analogies for how they work, but a lot less material about why we truly need them. Are there any alternatives or are these abstract concepts inevitable? 我们真的需要键（Keys）、查询（Queries）、值（Values）和点积注意力机制吗？互联网上流行的类比说：“你需要键和查询来让标记（tokens）之间进行对话。”但为什么呢？关于它们如何工作的类比有很多，但关于我们为何真正需要它们的资料却少得多。是否存在替代方案，还是说这些抽象概念是不可避免的？
This might seem like a silly question given the utter success of the Transformer architecture in 2026. But if history is any indication, Transformers will eventually be replaced by something better. The more we understand why they work, the faster we can move beyond them. 考虑到 Transformer 架构在 2026 年取得的巨大成功，这似乎是一个愚蠢的问题。但如果历史可以作为参考，Transformer 最终会被更好的东西所取代。我们越了解它们为何有效，就能越快地超越它们。
Incredibly, we can show that the general shape of the Transformer is hard to avoid! Start with a few key design pressures, and the alphabet soup of matrices begins to look much less arbitrary. Without relying on any “token asks a question” analogies, we’ll see that queries arise from a symmetry problem, while values and attention heads appear when we replace an unwieldy dynamic weight matrix with a small set of reusable transformations. 令人难以置信的是，我们可以证明 Transformer 的基本形态是难以避免的！从几个关键的设计压力出发，那些令人眼花缭乱的矩阵组合就开始变得不再那么随意了。在不依赖任何“标记提问”类比的情况下，我们将看到查询（Queries）源于对称性问题，而值（Values）和注意力头（Attention heads）的出现，是因为我们用一组可重用的转换替代了笨重的动态权重矩阵。
And as we reconcile our toy model with the original (autoregressive) Transformer, we’ll end with one last fascinating connection: the MLP—the often-overlooked feedforward block—can be viewed as a key-value store of its own. Ready to invent the Transformer for ourselves? First, we need to travel back to 2014 and ask why recurrent neural networks aren’t enough. 当我们把我们的简化模型与原始的（自回归）Transformer 进行调和时，我们将以最后一个迷人的联系作为结尾：MLP（通常被忽视的前馈模块）本身就可以被视为一个键值存储器。准备好亲自发明 Transformer 了吗？首先，我们需要回到 2014 年，探讨为什么循环神经网络（RNN）是不够的。
Why Fixed Memory Fails
为什么固定内存会失效
There are many issues with standard recurrent neural networks (RNNs.) Some issues, like the “gradient vanishing” issue, were solved by the extremely popular LSTM (Long Short-Term Memory network) by Hochreiter and Schmidhuber [1] which is a more advanced flavor of RNN. But one core issue remains for all flavors: RNNs “squish” past inputs together into fixed memory. 标准循环神经网络（RNN）存在许多问题。一些问题，例如“梯度消失”，通过 Hochreiter 和 Schmidhuber [1] 提出的极其流行的 LSTM（长短期记忆网络）得到了解决，这是一种更高级的 RNN。但所有变体都存在一个核心问题：RNN 将过去的输入“挤压”在一起，存入固定大小的内存中。
To see why, recall that an RNN has a memory state that is “written to” by the current input and the previous state. This connection between states in time is where the “recurrence” comes from. 为了理解原因，回想一下 RNN 有一个内存状态，它由当前的输入和之前的状态共同“写入”。这种时间状态之间的连接正是“循环”的来源。
Let’s use a crude analogy to show why this “fixed memory” is an issue. Say you encode the sentence “I have five dollars” into the RNN’s memory, which we’ll represent with the grey rectangle below. Now let’s extend that sentence to “I have five dollars and forty cents in my pocket.” 让我们用一个粗略的类比来说明为什么这种“固定内存”是个问题。假设你将句子“I have five dollars”（我有五美元）编码进 RNN 的内存中，我们用下面的灰色矩形来表示。现在，让我们把句子扩展为“I have five dollars and forty cents in my pocket”（我口袋里有五美元四十美分）。
Yikes. After adding more information into finite memory, there is greater “competition” for real estate and some memory is overwritten. This is disastrous when you need to recall specific facts or follow very specific instructions. 糟糕。在有限的内存中加入更多信息后，内存空间的“竞争”加剧，部分内存被覆盖了。当你需要回忆特定事实或遵循非常具体的指令时，这是灾难性的。
You might be thinking, “Why not use dynamic memory that grows with the input sequence?” Great intuition! This is exactly what Bahdanau et al. [2] tried in 2014 when they popularized the idea of “attention” within the RNN (yes, RNNs used attention before Transformers did!) 你可能会想：“为什么不使用随输入序列增长的动态内存呢？”直觉很棒！这正是 Bahdanau 等人 [2] 在 2014 年所尝试的，当时他们在 RNN 中普及了“注意力”的概念（没错，RNN 在 Transformer 出现之前就已经使用注意力机制了！）
The idea, at a high level, is to keep the entire previous history of the RNN states as our “expanding memory”, which naturally has the property of growing with the input sequence. 从宏观上看，这个想法是保留 RNN 状态的整个历史记录作为我们的“扩展内存”，它自然具备随输入序列增长的特性。
There is one big challenge that remains: training speed. At training time, we have each input sequence available up front. But generating the final Nth output requires N sequential steps in time given the recurrent dependencies; with long input sequences, we have long sequential computations that cannot be parallelized by GPUs. 但仍有一个巨大的挑战：训练速度。在训练时，我们可以预先获得每个输入序列。但由于循环依赖关系，生成最终的第 N 个输出需要 N 个时间步的顺序计算；对于长输入序列，我们会有很长的顺序计算过程，无法通过 GPU 进行并行化。
The idea in the landmark paper by Vaswani et al. [3] is this: What if we can remove those recurrent left-to-right connections? What if the red connections are all you need? Vaswani 等人 [3] 的里程碑式论文中的核心思想是：如果我们能移除那些从左到右的循环连接会怎样？如果那些红色的连接就是你所需要的一切呢？
Let’s stack the recurrent and non-recurrent architectures side by side, and compare their total number of compute steps by putting ordered labels for each step. Note how the non-recurrent model needed only 2 compute steps, since it can compute each layer entirely in parallel once the previous layer is computed. On the other hand, the recurrent model needed 5 compute steps due to the recurrent dependencies within each layer. 让我们将循环和非循环架构并排堆叠，并通过为每个步骤添加顺序标签来比较它们的总计算步数。请注意，非循环模型仅需 2 个计算步骤，因为它可以在计算完前一层后，完全并行地计算每一层。另一方面，由于每一层内部的循环依赖，循环模型需要 5 个计算步骤。
As the sequence gets longer, the non-recurrent model would stay at 2 steps while the recurrent model’s steps would grow forever with the sequence. It’s not looking good for recurrence! Let’s pivot to the non-recurrent model on the right. Now, we run into our next challenge: How on Earth should we pick the weights for these red connections? 随着序列变长，非循环模型仍保持在 2 个步骤，而循环模型的步骤会随着序列无限增长。这对循环模型来说不是好消息！让我们转向右侧的非循环模型。现在，我们遇到了下一个挑战：我们到底该如何为这些红色连接选择权重呢？
Transformers and Dynamic Weights
Transformer 与动态权重
If you look at the diag… 如果你看这张图……