What training actually does to an LLM

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

What training actually does to an LLM. The model you chat with was never written. Not one line of its behavior was composed by a person. Llama 3.1 8B is 8,006,464,512 numbers, counted on Meta’s own model card, and no human chose any of them. Everything it does, the grammar, the reasoning, the confident mistakes, is what those numbers happen to be. So where do the numbers come from? A loop, run billions of times. This is about that loop.

大语言模型（LLM）的训练究竟做了什么？你所对话的模型从未被“编写”过。它的行为中没有一行代码是由人撰写的。Llama 3.1 8B 模型包含 8,006,464,512 个数字（根据 Meta 官方模型卡统计），且没有人类选择过其中任何一个。它所做的一切——语法、推理、自信的错误——都仅仅是这些数字恰好呈现的状态。那么这些数字从何而来？来自一个运行了数十亿次的循环。本文正是关于这个循环。

The loop: Training starts with a pile of text and a model whose numbers are effectively random noise. The engine reads a chunk of text and predicts the next token, over and over. Then it compares each prediction with the token that was actually there and scores the miss. The score is called the loss; for language models it is usually cross-entropy loss, which punishes confident wrong guesses more than hesitant ones. Notice what the teacher is: the text itself. Nobody labels anything. The next token that actually follows is the label. That is why this is called self-supervised learning, and why it scales: the internet is the answer key.

循环：训练始于一堆文本和一个数字实际上是随机噪声的模型。引擎反复读取一段文本并预测下一个标记（token）。然后，它将每个预测与实际存在的标记进行比较，并对错误进行评分。这个分数被称为“损失”（loss）；对于语言模型，通常使用交叉熵损失，它对自信的错误猜测的惩罚力度要大于犹豫不决的猜测。请注意“老师”是谁：文本本身。没有人进行任何标注。紧随其后的下一个标记就是标签。这就是为什么它被称为自监督学习，也是它能够扩展的原因：互联网就是答案库。

The hard part: which knob did it? When the model is wrong, the error is one number at the output, but the model has billions of internal numbers. Which of them caused the miss? Every number is a weight, a dial that shapes what the next layer receives. Eight billion dials, one bad prediction. This is the credit assignment problem: eight billion suspects. The fix is older than deep learning. For each weight you want one signed number: the gradient, which answers “if I turn this dial up a little, does the error go up or down, and how fast?”

难点在于：是哪个旋钮导致的错误？当模型出错时，输出端只有一个错误数值，但模型内部却有数十亿个数字。究竟是哪一个导致了偏差？每个数字都是一个权重，一个决定下一层接收什么的旋钮。八十亿个旋钮，一个错误的预测。这就是“信用分配问题”：八十亿个嫌疑对象。解决办法比深度学习本身更古老。对于每个权重，你需要一个带符号的数字：梯度。它回答了这样一个问题：“如果我把这个旋钮调大一点，误差是增加还是减少，速度有多快？”

The weights are arranged in layers, each layer’s math feeding the next, and calculus has a rule for exactly this shape of problem: the chain rule, which says the sensitivity of the whole pipeline is the product of sensitivities along the path. Compute those sensitivities from the output end, layer by layer, reusing each layer’s arithmetic as you go, and you get every weight’s gradient for a small multiple of the cost of one forward pass. The error propagates backward through the network. Hence the name, backpropagation.

权重按层排列，每一层的计算结果传递给下一层。微积分中有一条专门针对此类问题的规则：链式法则。它指出整个流水线的敏感度等于路径上各部分敏感度的乘积。从输出端开始，逐层计算这些敏感度，并复用每一层的算术结果，你就能以仅相当于一次前向传播成本的微小倍数，获得每个权重的梯度。误差在网络中向后传播，因此得名“反向传播”（backpropagation）。

The 1986 Nature paper that made the method famous is literally titled “Learning representations by back-propagating errors” (Rumelhart, Hinton and Williams). History, briefly, because the 1986 paper did not invent it from nothing: reverse-mode automatic differentiation goes back to Linnainmaa (1970), and Paul Werbos applied it to multilayer networks in 1982. Rumelhart and colleagues rediscovered it independently, unaware of the earlier work, and their Nature paper is what the field remembers. The same line of ideas later earned Geoffrey Hinton a share of the 2024 Nobel Prize in Physics.

1986 年那篇让该方法成名的《自然》杂志论文，标题正是《通过反向传播误差来学习表征》（Rumelhart, Hinton 和 Williams 著）。简要回顾历史，因为 1986 年的论文并非凭空发明：反向模式自动微分可以追溯到 Linnainmaa（1970 年），而 Paul Werbos 在 1982 年将其应用于多层网络。Rumelhart 及其同事在不知晓前人工作的情况下独立重新发现了它，而他们的《自然》论文成为了该领域铭记的经典。同样的思想脉络后来为 Geoffrey Hinton 赢得了 2024 年诺贝尔物理学奖的一部分。

The step: Once every weight has its gradient, you move them all a little: new weight = old weight minus learning rate times gradient. The learning rate is the step size. Too big and the model jumps past the good regions and starts getting worse; too small and it crawls forever. This descent idea is old; gradient descent is generally attributed to Cauchy, 1847. Training is walking downhill in a space with billions of dimensions, where “downhill” means “the next word matches the text better.” One full pass, predict, score, backpropagate, step, is one update. You do it on a batch of text, then the next batch, and the next.

步骤：一旦每个权重都有了梯度，你就将它们全部移动一点：新权重 = 旧权重 - 学习率 × 梯度。学习率就是步长。如果太大，模型会跳过最优区域并开始变差；如果太小，模型会爬行得极其缓慢。这种下降的思想很古老；梯度下降法通常归功于 Cauchy（1847 年）。训练就是在拥有数十亿维度的空间中向下走，其中“向下”意味着“下一个词与文本匹配得更好”。一次完整的预测、评分、反向传播和更新步骤，构成了一次更新。你对一批文本执行此操作，然后是下一批，再下一批。

At scale, the documented case: GPT-3. The cleanest public record of this loop at full size is the GPT-3 paper (Brown et al., 2020). From its tables and appendix, all directly quotable: 175 billion weights, arranged in 96 layers. Each update sees a batch of 3.2 million tokens. Training ran for 300 billion tokens total. 300 billion divided by 3.2 million is about 94,000 updates. The model’s whole education was 94,000 nudges, each nudge adjusting 175 billion dials.

在大规模应用中，有据可查的案例是 GPT-3。关于这一循环在全尺寸下运行的最清晰公开记录是 GPT-3 的论文（Brown 等人，2020 年）。从其表格和附录中可以直接引用：1750 亿个权重，排列在 96 层中。每次更新处理 320 万个标记的批次。训练总共运行了 3000 亿个标记。3000 亿除以 320 万约为 94,000 次更新。该模型的全部教育过程就是 94,000 次微调，每一次微调都在调整 1750 亿个旋钮。

Between the 1986 paper and 2020, the loop gained instruments, all documented in the same appendix: Adam optimizer (beta1 = 0.9, beta2 = 0.95): keeps a running memory of each weight’s history and scales each step individually. Momentum plus a per-dial step size, instead of one fixed step for everyone. Gradient clipping at 1.0: if one batch produces an absurdly large gradient, it gets capped. One ugly batch cannot shove the whole model sideways. Learning rate schedule: a warmup over the first 375 million tokens, then a cosine decay down to 10% of the peak over 260 billion tokens. Confident, careful steps early; small steps later. Weight decay of 0.1, a mild pressure that keeps weights from growing without bound.

在 1986 年的论文和 2020 年之间，这个循环增加了许多工具，全部记录在同一附录中：Adam 优化器（beta1 = 0.9, beta2 = 0.95）：保留每个权重历史的运行记忆，并单独缩放每一步。这是动量加上每个旋钮的步长，而不是对所有人使用固定的步长。梯度裁剪（Gradient clipping）设为 1.0：如果某一批次产生异常大的梯度，它会被限制住。一个糟糕的批次不会让整个模型偏离轨道。学习率调度：在前 3.75 亿个标记上进行预热，然后在 2600 亿个标记上进行余弦衰减，降至峰值的 10%。早期步子自信而谨慎，后期步子则很小。0.1 的权重衰减，这是一种轻微的压力，防止权重无限制地增长。

Modern practice is the same loop with better instruments. Llama 2 (2023) used AdamW with beta1 = 0.9, beta2 = 0.95, weight decay 0.1, gradient clipping at 1.0, and was pretrained on 2 trillion tokens. The Llama 3 flagship (2024) ran the loop on 15.6 trillion tokens at roughly 50 times the compute of the largest Llama 2. The 1986 recipe, industrialized.

现代实践依然是同一个循环，只是工具更先进了。Llama 2（2023 年）使用了 AdamW，参数为 beta1 = 0.9, beta2 = 0.95，权重衰减 0.1，梯度裁剪 1.0，并在 2 万亿个标记上进行了预训练。Llama 3 旗舰版（2024 年）在 15.6 万亿个标记上运行了该循环，计算量约为最大版 Llama 2 的 50 倍。这就是 1986 年配方的工业化版本。

What you get: After all those updates, no single weight means anything. There is no dial labeled “knows the capital of France.” A fact, a rule, a style, all of it is smeared across billions of weights, and it only exists when the whole network runs. This is the same point as the earlier piece on hallucinations: knowledge in an LLM is compressed into weights, not filed anywhere. Training does not install facts. It tunes the model until its next-token guesses statistically match the text it read. That is also why the model cannot check its own answers. There is no lookup table to consult, no certificate of correctness, only the pattern. And it is why the confident mistakes happen: the tuning optimizes for text that sounds right, and sounding right is not the same as being right.

你得到了什么：经过所有这些更新后，没有任何单个权重具有特定含义。没有一个旋钮被标记为“知道法国的首都是哪里”。事实、规则、风格，所有这些都分散在数十亿个权重中，只有当整个网络运行时，它们才存在。这与之前关于幻觉的文章观点一致：LLM 中的知识被压缩在权重中，而不是存储在任何地方。训练并不会“安装”事实。它只是调整模型，直到其对下一个标记的猜测在统计上与它所阅读的文本相匹配。这也是模型无法检查自己答案的原因。没有可供查询的查找表，没有正确性证书，只有模式。这也是为什么会发生自信的错误：调整过程优化的是“听起来正确”的文本，而“听起来正确”并不等同于“事实正确”。

So, the whole story: an LLM is not written or taught. It is tuned. Random noise, turned 94,000 times (GPT-3 scale) or fed 15.6 trillion tokens of text (Llama 3 scale), until prediction and reality line up. The results are remarkable, and the mechanism is a loop so plain that a 1986 paper title sti

总结全文：大语言模型既不是被编写出来的，也不是被教导出来的。它是被“调优”出来的。随机噪声经过 94,000 次（GPT-3 规模）的旋转，或被喂入 15.6 万亿个标记的文本（Llama 3 规模），直到预测与现实对齐。结果是惊人的，而其机制却是一个如此简单的循环，以至于 1986 年那篇论文的标题至今……