From “token” to “MoE”: the LLM glossary in dependency order

从“Token”到“MoE”：按依赖顺序排列的 LLM 术语表

TL;DR — A glossary to actually understand the terms you hit when reading about LLMs: token, embedding, attention, KV cache, GQA, MoE, quantization and the rest. But not alphabetical — in dependency order: every entry uses only concepts already explained above, so if you read it start to finish, by the time you reach “Mixture of Experts” you already have all the pieces to get it. 简而言之——这是一份旨在让你真正理解阅读 LLM 相关文章时所遇到术语的词汇表：Token、Embedding、Attention、KV Cache、GQA、MoE、量化等。它不是按字母顺序排列的，而是按依赖顺序排列的：每个条目仅使用前面已经解释过的概念。因此，如果你从头读到尾，当你读到“混合专家模型（Mixture of Experts）”时，你已经掌握了理解它所需的所有知识碎片。

The analogies come from the world of the people who hang around this blog: networks, caches, PID controllers, tuning. No math prerequisite beyond “I know what an array is.” 文中的类比来自本博客读者的背景领域：网络、缓存、PID 控制器、调优。除了“知道什么是数组”之外，没有任何数学先修要求。

Alphabetical LLM glossaries have a structural flaw: the “attention” entry sends you to “softmax”, which is further down, which in turn uses “logits”, which is earlier but assumes “vocabulary”. You end up hopping around like a Wikipedia page at two in the morning. 按字母顺序排列的 LLM 词汇表存在结构性缺陷：“Attention”条目会把你引向排在后面的“Softmax”，而“Softmax”又会用到更靠前的“Logits”，但“Logits”又预设了你已经理解“Vocabulary”。最终你会像凌晨两点刷维基百科一样，在各个页面间跳来跳去。

This document is built the other way around: like a chain. It’s the glossary I wish I’d had, and the analogies are the ones that made the concepts click for me — someone coming from sysadmin, networks and controllers, not a PhD in ML. If you too think in terms of caches, error signals and control loops, you’re in the right place. 本文档采用相反的方式构建：像一条链条。这是我一直希望拥有的词汇表，其中的类比让我——一个来自系统管理、网络和控制器领域，而非机器学习博士背景的人——真正理解了这些概念。如果你也习惯用缓存、错误信号和控制回路来思考，那么你来对地方了。

A note of honesty, before we start. This glossary doesn’t come from knowledge I already possess. It comes from the questions I asked an LLM — “explain the KV cache to me”, “why that √d?” — and from the explanations it wrote for me. Some sank in, others I’m still chewing on: I don’t fully master everything you read here, and I won’t pretend to. 在开始之前，先诚实地说明一下：这份词汇表并非源于我已有的知识。它源于我向 LLM 提出的问题——“向我解释一下 KV Cache”、“为什么是 √d？”——以及它为我撰写的解释。有些内容我已经消化了，有些我还在琢磨：我并没有完全掌握这里提到的所有内容，我也不会假装自己掌握了。

What I needed was a single place to come back to — a notebook to reopen when I run into one of these terms while reading something on the topic, to reread it slowly and with the analogies that pin it down for me. I’ve left it here in case it’s useful to you too; but it is, first and foremost, mine. 我需要的是一个可以随时回溯的地方——一个当我阅读相关主题遇到这些术语时可以重新打开的笔记本，通过那些能帮我定格概念的类比，慢慢重读。我把它留在这里，以防它对你也有用；但它首先且最主要是属于我自己的。

To keep our feet on the ground I’ll use as a throughline the numbers of a small but modern-architecture model: 180 million parameters, a 32,768-token vocabulary, internal size 640, 20 layers. Small enough to train on human hardware, modern enough to contain every concept that matters in 2026. 为了保持脚踏实地，我将以一个小型但架构现代的模型参数作为贯穿始终的基准：1.8 亿参数、32,768 个 Token 的词汇表、640 的内部维度、20 层网络。它小到可以在个人硬件上训练，又现代到足以涵盖 2026 年所有重要的概念。

Level 0 — What an LLM actually does

第 0 级 — LLM 到底在做什么

Language modeling The whole game is ONE thing: given a sequence of text, predict the next piece. “The cat is on the” → probably “roof” or “couch”, improbably “carburetor”. An LLM is a giant function that takes text in and returns a probability distribution over the next piece. Everything else — chat, reasoning, code — emerges from this task repeated billions of times over human text. When it “generates”, the model predicts a piece, appends it to the input, predicts the next, and so on. Like autocomplete taken to an absurd extreme. 语言建模 整个游戏的本质只有一件事：给定一段文本序列，预测下一个片段。“The cat is on the” → 可能是“roof”（屋顶）或“couch”（沙发），不可能是“carburetor”（化油器）。LLM 是一个巨大的函数，它接收文本输入，并返回下一个片段的概率分布。其他一切——聊天、推理、代码——都是通过在人类文本上重复数十亿次这一任务而涌现出来的。当它“生成”时，模型预测一个片段，将其附加到输入中，再预测下一个，依此类推。就像被推向荒谬极致的自动补全。

Token The “piece” above. Not a word nor a character: something in between, a frequent fragment. “computing” might be a single token; “quadcopter” might split into “quad”+“cop”+“ter”. The tokenizer is the component that converts text ↔ sequences of integers (each token has an ID). Think of a lookup table: token 4521 = ” cat”. The vocabulary is the set of all known tokens — in our reference model, 32,768. Analogy: it’s a form of dictionary compression, conceptually a cousin of LZ — frequent sequences become single symbols. Token 上述的“片段”。它既不是单词也不是字符：而是介于两者之间，即一个高频片段。“computing”可能是一个 Token；“quadcopter”可能会拆分为“quad”+“cop”+“ter”。Tokenizer（分词器）是将文本与整数序列进行转换的组件（每个 Token 都有一个 ID）。可以把它想象成一张查找表：Token 4521 = “ cat”。词汇表（Vocabulary）是所有已知 Token 的集合——在我们的参考模型中，有 32,768 个。类比：这是一种字典压缩形式，概念上是 LZ 算法的表亲——高频序列变成了单个符号。

BPE (Byte-Pair Encoding) The algorithm that builds that dictionary: in the modern byte-level variant, you start from single bytes, find the most frequent adjacent pair in the corpus, merge it into a new symbol, and repeat until you reach the desired vocabulary size. Purely statistical, no linguistic magic. BPE (字节对编码) 构建该字典的算法：在现代字节级变体中，你从单个字节开始，找出语料库中最频繁出现的相邻对，将其合并为一个新符号，然后重复此过程，直到达到所需的词汇表大小。纯粹是统计学，没有任何语言学魔法。

Level 1 — The mathematical building blocks (the bare minimum)

第 1 级 — 数学基石（最简要求）

Vector, matrix, tensor A vector is an array of floats. A matrix is a 2D array. A tensor is the generalization to N dimensions — in PyTorch code you see shapes like (B, T, D) = (batch, positions in the sequence, dimensions per position). Nothing mystical: multidimensional arrays with vectorized operations, like NumPy. 向量、矩阵、张量 向量是浮点数数组。矩阵是二维数组。张量是向 N 维的推广——在 PyTorch 代码中，你会看到像 (B, T, D) 这样的形状 = (批次, 序列中的位置, 每个位置的维度)。没什么神秘的：就是带有向量化操作的多维数组，就像 NumPy 一样。

Matrix-vector multiplication = transformation The fundamental operation of ALL deep learning: y = W @ x. Take a vector x, multiply it by a matrix W of weights, get a new vector y. Each element of y is a weighted combination of all the elements of x. A matrix = one learned transformation from one space to another. An LLM is, brutally, hundreds of these multiplications in a row with a bit of non-linearity in between. When you read “linear layer” or nn.Linear: this is it. When you read GEMM: General Matrix Multiply, the kernel GPUs grind on. 矩阵-向量乘法 = 变换 所有深度学习的基本操作：y = W @ x。取一个向量 x，将其乘以权重矩阵 W，得到一个新的向量 y。y 的每个元素都是 x 中所有元素的加权组合。矩阵 = 从一个空间到另一个空间的一种学习到的变换。粗暴地说，LLM 就是连续进行数百次这样的乘法，中间夹杂着一点非线性。当你读到“线性层（linear layer）”或 nn.Linear 时：指的就是这个。当你读到 GEMM 时：通用矩阵乘法（General Matrix Multiply），这是 GPU 疯狂计算的核心内核。

Parameters (or weights) The numbers inside those matrices. “180M-parameter model” = the sum of all elements of all matrices is 180 million floats. It’s the parameters that get modified during training — the model’s “knowing” lives entirely there. The architecture is the printed circuit board; the parameters are the component values that training solders on. 参数（或权重） 矩阵内部的数字。“1.8 亿参数模型” = 所有矩阵中所有元素的总和为 1.8 亿个浮点数。在训练过程中被修改的就是这些参数——模型的“知识”完全存在于这里。架构是印刷电路板；参数则是训练过程焊接上去的组件数值。

Embedding The bridge between tokens (integers) and math (vectors). A table: row 4521 = the 640-float vector representing ” cat”. The deep part: during training these vectors organize themselves geometrically by meaning — “cat” and “feline” end up close in this 640-dimensional space, “cat” and “lathe” far apart. The direction itself encodes semantic relations. If you’ve ever used pgvector or a vector database for semantic search, it’s the exact same concept — except here the embeddings are internal to the model and learned along with everything else. Embedding（嵌入） Token（整数）与数学（向量）之间的桥梁。一张表：第 4521 行 = 代表“ cat”的 640 个浮点数的向量。深层含义：在训练过程中，这些向量会根据含义在几何上进行自我组织——“cat”和“feline”最终在这个 640 维空间中靠得很近，“cat”和“lathe”则相距甚远。方向本身编码了语义关系。如果你曾经使用过 pgvector 或向量数据库进行语义搜索，概念完全一样——只是这里的 Embedding 是模型内部的，并且是与其他所有内容一起学习出来的。

d_model The model’s “width”: how many floats represent each token as it flows through the network. In our reference, 640. Each token enters as a 640-vector, exits every layer as a 640-vector (enriched with context), all the way to the end. It’s the system’s data bus. d_model 模型的“宽度”：当每个 Token 流经网络时，用多少个浮点数来表示它。在我们的参考模型中是 640。每个 Token 进入时是一个 640 维向量，经过每一层后输出的依然是一个 640 维向量（包含更丰富的上下文），直到最后。它是系统的“数据总线”。

Activation function (non-linearity) After a matrix multiplication you apply a non-linear function element by element (ReLU, SiLU, GELU — variants on the same theme: squash or let through). Why it’s needed: a chain of purely linear operations mathematically collapses into ONE linear operation — you’d learn nothing complex. The non-linearity is what lets the network represent arbitrary functions. 激活函数（非线性） 在矩阵乘法之后，你逐元素应用一个非线性函数（ReLU、SiLU、GELU——同一主题的变体：压缩或放行）。为什么需要它：纯线性操作的链条在数学上会坍缩成“一个”线性操作——你将学不到任何复杂的东西。非线性是让网络能够表示任意函数的关键。