Don’t Just “Throw Adam at It”: Misunderstanding Adam Will Cost You

查看原文 / View Original 编辑 / Edit

Don’t Just “Throw Adam at It”: Misunderstanding Adam Will Cost You

别再只会“无脑套用 Adam”：误解 Adam 将让你付出代价

You “vibe-coded” the import. Understand Adam’s optimization dynamics, why it fails spectacularly, and how to fix it. 你只是凭“感觉”导入了优化器。你需要理解 Adam 的优化动力学、它为何会惨败，以及如何修复它。

Optimization landscapes aren’t always so theoretically neat. 优化地形并不总是像理论上那样整洁。

I spent weeks working on a particularly hard reinforcement learning problem. The research demonstrated similar agents learning comparable tasks. Yet our model was dead in the water. I simplified the architecture. I added layers. I removed layers. I swapped LSTMs for Transformers, added attention, removed it. I rebuilt the input features at least 20 times. I even tried exotic memory architectures. I spent thousands on GPU hours in pathological hyperparameter search mode and I couldn’t get the results. Nada. 我花了数周时间处理一个特别困难的强化学习问题。研究表明，类似的智能体可以学习相当的任务，但我们的模型却毫无进展。我简化了架构，增加了层数，又删减了层数。我将 LSTM 换成了 Transformer，添加了注意力机制，又将其移除。我重构了输入特征至少 20 次，甚至尝试了奇特的记忆架构。我花费了数千美元的 GPU 时长进行病态的超参数搜索，却依然一无所获。

The fix, when I finally found it, was humiliating: I changed β₂ from 0.999 to 0.95 and reduced β₁ from 0.9 to 0.5. 当我最终找到解决方案时，结果令人汗颜：我将 β₂ 从 0.999 改为 0.95，并将 β₁ 从 0.9 降低到 0.5。

# The "humiliating" fix that actually worked:
# 那个真正奏效的“令人汗颜”的修复方案：
optimizer = optim.AdamW(
    model.parameters(),
    lr=3e-4,
    betas=(0.5, 0.95), # The magic numbers / 神奇的数字
    eps=1e-4 # Preventing division by zero in RL / 防止强化学习中的除零错误
)

Most deep learning engineers, myself included, just do the following: Just use AdamW with the default parameters, and set the learning rate to 3e-4. I don’t vibe-code deep learning models out of personal preference. If you do, you will most certainly see this as the optimizer choice and learning rate. 大多数深度学习工程师（包括我）通常的做法是：直接使用带有默认参数的 AdamW，并将学习率设为 3e-4。我并不是出于个人偏好才凭“感觉”编写深度学习模型。如果你也这样做，你几乎肯定会选择这个优化器和学习率。

In fact, let’s test GPT 5.6 with the prompt: “create a training script for a deep learning model” 事实上，让我们用提示词“创建一个深度学习模型的训练脚本”来测试一下 GPT 5.6。

Unsurprisingly, GPT happily spits out: 不出所料，GPT 愉快地输出了：

CONFIG = {
    "lr": 3e-4,
    "weight_decay": 1e-5,
    ...
}
....
optimizer = optim.AdamW(
    model.parameters(),
    lr=CONFIG["lr"],
    weight_decay=CONFIG["weight_decay"]
)

“In Karpathy we trust. 3e-4 is the best learning rate for Adam, hands down.”— Andrej Karpathy (@karpathy) November 24, 2016 “我们相信 Karpathy。毫无疑问，3e-4 是 Adam 最好的学习率。”—— Andrej Karpathy (@karpathy) 2016年11月24日

Nope. The problem with this dangerous assumption, is that engineers glaze over this very important aspect of training, treating this part of the equation as solved. You may be able to get away with it, as it provides a sensible default, but in the long run, especially when encountering a non-stationary or difficult optimization landscape, you will fail. Hard. 不对。这种危险假设的问题在于，工程师们往往会忽略训练中这一极其重要的方面，将方程的这一部分视为“已解决”。你或许能侥幸过关，因为它提供了一个合理的默认值，但从长远来看，尤其是在遇到非平稳或困难的优化地形时，你会惨败。

In this article, I’ll cover: How Adam actually works, Where Adam fails spectacularly, The intuition behind modifying the defaults. This isn’t another generic review of parameter optimization. This is why your incorrect assumption of “just use Adam” is wrong with the mathematical intuition as to why. 在本文中，我将涵盖：Adam 的实际工作原理、Adam 在何处惨败、修改默认参数背后的直觉。这不是另一篇关于参数优化的通用综述，而是通过数学直觉解释为什么你“直接用 Adam 就行”的假设是错误的。

Who’s this for: You use Adam as your optimizer of choice with a surface level (or no) understanding of how it works and where it fails. 适用人群：将 Adam 作为首选优化器，但对其工作原理和失效场景仅有浅显（或完全没有）了解的人。

How Adam actually works

Adam 的实际工作原理

Vanilla stochastic gradient descent (SGD) applies the same learning rate to every parameter: 原始的随机梯度下降 (SGD) 对每个参数应用相同的学习率：

$\theta_t = \theta_{t-1} – \alpha \cdot g_t$

This is the equation most practitioners are familiar with. If you have a CS degree, you may have implemented it from scratch. Multiply $\alpha$ (the learning rate) by the $g_t$ (gradient) and voila, you’ve got an update. This has some fundamental problems, as heavily documented in existing research, including fundamental instability caused by noisy gradients. Everybody knows this, which is why SGD is mostly treated like a second rate citizen*. 这是大多数从业者熟悉的方程。如果你有计算机科学学位，你可能从零实现过它。将 $\alpha$（学习率）乘以 $g_t$（梯度），瞧，你就得到了一个更新。正如现有研究中所记录的那样，这存在一些根本性问题，包括由噪声梯度引起的根本性不稳定。大家都知道这一点，这就是为什么 SGD 通常被视为“二等公民”*。

Variants soon were invented: SGD with Momentum, RMSProp, etc. 随后发明了各种变体：带动量的 SGD、RMSProp 等。

  • I say mostly because SGD is sometimes used as the first choice in large scale image problems.
  • 我说“通常”是因为在某些大规模图像问题中，SGD 有时仍是首选。

In comes Adam. Adam (Kingma & Ba, 2015) was designed to solve both problems simultaneously: smooth noisy gradients and adapt step sizes per parameter based on observed gradient statistics. Adam maintains two running statistics for each parameter in your model, which can be mathematically elegant but means triple the memory requirements with gigantic state_dicts. Adam 登场了。Adam (Kingma & Ba, 2015) 的设计初衷是同时解决这两个问题：平滑噪声梯度，并根据观测到的梯度统计信息为每个参数调整步长。Adam 为模型中的每个参数维护两个运行统计量，这在数学上很优雅，但意味着对于巨大的 state_dicts，内存需求会增加三倍。

You may have seen this 1000 times. But do you actually understand it? Read this thoroughly until it is understood. 你可能见过这个公式 1000 次了，但你真的理解它吗？请仔细阅读，直到完全理解。

First moment estimate (the momentum): 一阶矩估计（动量）：

$m_t = \beta_1 \cdot m_{t-1} + (1 – \beta_1) \cdot g_t$

We need to understand each variable in this equation. 我们需要理解这个方程中的每个变量。

  • $m_t$: The exponentially weighted average of gradients up to the current timestep $t$. -> This is the actual direction applied to the model params.
  • $m_t$：截至当前时间步 $t$ 的梯度指数加权平均值。-> 这是应用于模型参数的实际方向。
  • $\beta_1$: the decay of the momentum, set to 0.9 as a default. This is what you can control.
  • $\beta_1$：动量的衰减率，默认设为 0.9。这是你可以控制的参数。
  • $g_t$: the calculated gradient for the timestamp. -> This is not applied to the model.
  • $g_t$：当前时间戳计算出的梯度。-> 这并没有直接应用于模型。

So the gradient applied for each timestep has accumulated directional information. It holds onto the average direction, instead of using the direction calculated for each timestep. You must understand the actual gradient is never applied. It only directionally affects the final gradient, which may be quite different. Your actual gradient only perturbs the accumulated momentum, which is the direction in which the gradient actually moves. 因此，每个时间步应用的梯度都积累了方向信息。它保留了平均方向，而不是使用每个时间步计算出的方向。你必须明白，实际的梯度从未被直接应用。它只是在方向上影响最终的梯度，而最终梯度可能大不相同。你的实际梯度只是扰动了累积动量，而这才是梯度实际移动的方向。

Second moment estimate (the variance): 二阶矩估计（方差）：

$v_t = \beta_2 \cdot v_{t-1} + (1 – \beta_2) \cdot g_t^2$

  • $v_t$: The exponentially weighted average of the squared gradient up to the current timestep $t$. -> This determines how the update is normalized.
  • $v_t$：截至当前时间步 $t$ 的梯度平方的指数加权平均值。-> 这决定了更新如何被归一化。
  • $\beta_2$: the decay of the squared gradient, set to 0.999 as a default. This is what you can control.
  • $\beta_2$：梯度平方的衰减率，默认设为 0.999。这是你可以控制的参数。
  • $g_t^2$: the calculated squared gradient for the timestamp.
  • $g_t^2$：当前时间戳计算出的梯度平方。

This second moment controls the actual step size. It’s not like the learning rate doesn’t matter, but Adam effectively bumps up the applied gradient for parameters which typically receive a small actual gradient and dampens the applied gradient for those which receive large gradients. $v_t$ controls how much the gradient is normalized per step. Large gradients dampen the step size. Small gradients can increase the step size. 这个二阶矩控制着实际的步长。并不是说学习率不重要，但 Adam 有效地为那些通常接收到较小实际梯度的参数增加了应用梯度，并为那些接收到较大梯度的参数抑制了应用梯度。$v_t$ 控制每一步梯度被归一化的程度。大梯度会抑制步长，小梯度则可以增加步长。

Note that the default here: 0.999 averages over the last ~1000 steps. 注意这里的默认值：0.999 意味着对过去约 1000 个步骤进行平均。

Bias correction 偏差修正

Because $m_0 = 0$ and $v_0 = 0$, the estimates are biased toward zero in early training. Adam corrects for this: 由于 $m_0 = 0$ 且 $v_0 = 0$，这些估计值在训练初期会向零偏移。Adam 对此进行了修正：

$\hat{m}_t = \frac{m_t}{1 – \beta_1^t} \qquad \hat{v}_t = \frac{v_t}{1 – \beta_2^t}$

← 返回新闻列表 / Back to news list