Backpropagation Explained for Beginners (Part 2): There Has to Be a Better Way
Backpropagation Explained for Beginners (Part 2): There Has to Be a Better Way
深度学习反向传播入门（第二部分）：一定有更好的方法
The idea that makes backpropagation possible. 这是使反向传播成为可能的核心思想。
Welcome back! Let’s continue our journey of understanding backpropagation in detail. Let’s briefly recall what we covered in Part 1 so far. 欢迎回来！让我们继续深入探索反向传播的奥秘。首先，让我们简要回顾一下第一部分的内容。
A Quick Recap / 快速回顾
We first started trying to understand neural networks using a simple dataset. In that process, we first built a small neural network and understood how it makes predictions through the process of forward propagation. 我们最初尝试通过一个简单的数据集来理解神经网络。在此过程中，我们构建了一个小型神经网络，并了解了它是如何通过前向传播过程进行预测的。
We then observed that the predicted values were nowhere close to the actual values, resulting in a large error. Now, we wanted to train this neural network to perform better. By comparing it with simple linear regression, we observed that, in our neural network, the loss depends on seven parameters: $[w_1, w_2, w_3, w_4, b_1, b_2, b_3]$. 随后我们观察到，预测值与实际值相差甚远，导致了较大的误差。为了让神经网络表现更好，我们需要对其进行训练。通过与简单的线性回归进行对比，我们发现神经网络的损失函数取决于七个参数：$[w_1, w_2, w_3, w_4, b_1, b_2, b_3]$。
Our complete loss function looked like this: 我们完整的损失函数如下所示： $$ L(w_1,w_2,w_3,w_4,b_1,b_2,b_3)=\frac{1}{n}\sum_{i=1}^{n}\left(y_i-\left(w_3\mathrm{ReLU}(w_1x_i+b_1)+w_4\mathrm{ReLU}(w_2x_i+b_2)+b_3\right)\right)^2 $$
Next, just as we did in simple linear regression, we wanted to differentiate the loss with respect to each parameter. We started with $w_1$ and used the classical differentiation method. Eventually, we arrived at the following result: 接下来，就像我们在简单线性回归中所做的那样，我们希望对每个参数求损失函数的导数。我们从 $w_1$ 开始，使用经典的微分法，最终得到了以下结果： $$ \frac{\partial L}{\partial w_1} = -\frac{2}{n} \sum_{i=1}^{n} (y_i-\hat{y}_i) w_3 \mathrm{ReLU}‘(w_1x_i+b_1) x_i $$
Do We Really Have to Repeat This? / 我们真的必须重复这个过程吗？
Now it’s time to think. We differentiated the loss with respect to $w_1$, which means we were trying to find how the loss changes as $w_1$ changes. We used the classical differentiation method here. We now have six more parameters to differentiate the loss with respect to. What do you think? Do we need to proceed with the same method? No. 现在是时候思考一下了。我们对 $w_1$ 求了损失函数的导数，这意味着我们试图找出当 $w_1$ 变化时损失是如何变化的。我们在这里使用了经典的微分法。现在还有六个参数需要求导，你觉得呢？我们需要继续使用同样的方法吗？不需要。
The Chain Rule to the Rescue / 链式法则来救场
If you remember, we discussed the chain rule in Part 1. We used a simple example. We considered $y=x^2$ and $z=y^3$. We noticed that $z$ does not depend directly on $x$. Instead, the relationship looks like $x \rightarrow y \rightarrow z$. 如果你还记得，我们在第一部分讨论过链式法则。我们使用了一个简单的例子：考虑 $y=x^2$ 和 $z=y^3$。我们注意到 $z$ 并不直接依赖于 $x$，它们的关系是 $x \rightarrow y \rightarrow z$。
This means that when $x$ changes, it first changes $y$, and the change in $y$ then affects $z$. Since the effect of $x$ reaches $z$ through an intermediate variable, we cannot differentiate $z$ with respect to $x$ directly. Instead, we used the chain rule: 这意味着当 $x$ 发生变化时，它首先会改变 $y$，而 $y$ 的变化进而影响 $z$。由于 $x$ 的影响是通过中间变量传递给 $z$ 的，我们不能直接对 $x$ 求 $z$ 的导数。相反，我们使用了链式法则： $$ \frac{dz}{dx} = \frac{dz}{dy} \cdot \frac{dy}{dx} $$
The chain rule tells us that instead of taking one large step from $x$ to $z$, we can break the problem into smaller steps by following the dependency path. 链式法则告诉我们，与其从 $x$ 到 $z$ 跨越一大步，不如沿着依赖路径将问题分解为更小的步骤。
Following the Dependency Path / 沿着依赖路径
Now the question is can we apply this same idea to our neural network. The answer is yes. To do that, we first need to understand how a change in $w_1$ travels through the neural network before it finally affects the loss. Let’s write down the equations of our neural network. 现在的问题是，我们能否将同样的思想应用到神经网络中？答案是肯定的。为此，我们首先需要了解 $w_1$ 的变化是如何在神经网络中传递，并最终影响损失的。让我们写出神经网络的方程：
$z_1=w_1x+b_1$ ($z_1$ depends on the weight $w_1$, the input $x$, and the bias $b_1$) $z_1=w_1x+b_1$（$z_1$ 依赖于权重 $w_1$、输入 $x$ 和偏置 $b_1$）
Next, we applied the ReLU activation function: $a_1=\mathrm{ReLU}(z_1)$. Here, the activation $a_1$ depends entirely on $z_1$. 接下来，我们应用 ReLU 激活函数：$a_1=\mathrm{ReLU}(z_1)$。这里，激活值 $a_1$ 完全取决于 $z_1$。
The prediction of our network is $\hat{y}=w_3a_1+w_4a_2+b_3$. We can see that, the prediction depends on the activation $a_1$. Finally, we computed the loss: $L=(y-\hat{y})^2$. The loss depends on the prediction. 我们网络的预测值为 $\hat{y}=w_3a_1+w_4a_2+b_3$。我们可以看到，预测值取决于激活值 $a_1$。最后，我们计算损失：$L=(y-\hat{y})^2$。损失取决于预测值。
If we connect all these relationships together, we get: $w_1 \rightarrow z_1 \rightarrow a_1 \rightarrow \hat{y} \rightarrow L$. 如果我们把所有这些关系连接起来，就得到了：$w_1 \rightarrow z_1 \rightarrow a_1 \rightarrow \hat{y} \rightarrow L$。
Comparing this with the simple example we considered: 将此与我们之前考虑的简单例子进行对比： Simple example: $x \rightarrow y \rightarrow z$ 简单例子：$x \rightarrow y \rightarrow z$ Neural network: $w_1 \rightarrow z_1 \rightarrow a_1 \rightarrow \hat{y} \rightarrow L$ 神经网络：$w_1 \rightarrow z_1 \rightarrow a_1 \rightarrow \hat{y} \rightarrow L$
As we already said in part 1, the only difference is that the neural network has a longer dependency chain, but the underlying idea remains the same. 正如我们在第一部分所说，唯一的区别在于神经网络拥有更长的依赖链，但其核心思想是一样的。
Partial Derivatives vs. the Chain Rule / 偏导数与链式法则
At this point, we may get confused between partial derivatives and the chain rule. Let’s clarify both ideas before moving forward. 此时，我们可能会混淆偏导数和链式法则。在继续之前，让我们澄清这两个概念。
In the simple example, $y=x^2$, $y$ is only depended on input variable, $x$. So, to find change in $y$ w.r.t $x$, ordinary derivatives are sufficient, and we write $\frac{dy}{dx}$. 在简单例子 $y=x^2$ 中，$y$ 仅依赖于输入变量 $x$。因此，要找出 $y$ 相对于 $x$ 的变化，普通导数就足够了，我们写作 $\frac{dy}{dx}$。
But, consider the equation $z_1=w_1x+b_1$. Here $z_1$ depends on three variables: $w_1$, $x$, and $b_1$. Now, we want to know how $z_1$ changes when only $w_1$ changes. For this, we temporarily keep $x$ and $b_1$ fixed. Therefore, we write $\frac{\partial z_1}{\partial w_1}$. 但是，考虑方程 $z_1=w_1x+b_1$。这里 $z_1$ 依赖于三个变量：$w_1$、$x$ 和 $b_1$。现在，我们想知道当只有 $w_1$ 变化时 $z_1$ 如何变化。为此，我们暂时保持 $x$ 和 $b_1$ 不变。因此，我们写作 $\frac{\partial z_1}{\partial w_1}$。
The symbol $\partial$ tells us that we are changing only one variable while keeping the remaining variables as constants during that particular calculation. This is what we know about partial derivatives. 符号 $\partial$ 告诉我们，在特定的计算过程中，我们只改变一个变量，而保持其余变量为常数。这就是我们所理解的偏导数。
Now, coming to the chain rule, it is a different concept. Let’s be careful not to confuse these two different concepts. A partial derivative answers the question: Which variable are we changing? For example, $\frac{\partial z_1}{\partial w_1}$ which means only $w_1$ is allowed to change. 现在谈到链式法则，这是一个不同的概念。我们要小心，不要混淆这两个不同的概念。偏导数回答的问题是：我们正在改变哪个变量？例如，$\frac{\partial z_1}{\partial w_1}$ 意味着只允许 $w_1$ 发生变化。
The chain rule answers a different question: How does the effect of one variable travel through several intermediate computations? In our neural network, $w_1 \rightarrow z_1 \rightarrow a_1 \rightarrow \hat{y} \rightarrow L$. As the effect of changing $w_1$ reaches the loss through every intermediate parameter, we multiply the derivatives along this path. 链式法则回答的是另一个问题：一个变量的影响是如何通过多个中间计算传递的？在我们的神经网络中，$w_1 \rightarrow z_1 \rightarrow a_1 \rightarrow \hat{y} \rightarrow L$。由于改变 $w_1$ 的影响通过每一个中间参数传递到损失函数，我们将这条路径上的导数相乘。
Therefore: 因此： $$ \frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial a_1} \cdot \frac{\partial a_1}{\partial z_1} \cdot \frac{\partial z_1}{\partial w_1} $$
We can observe that nothing new has happened. We are still applying the same chain rule that we learned in Part 1. The only difference is that the chain is now longer and the functions involve multiple variables, which is why we use partial derivatives instead of ordinary derivatives. 我们可以观察到，并没有什么新鲜事发生。我们仍然在应用第一部分学到的链式法则。唯一的区别是链条变长了，且函数涉及多个变量，这就是为什么我们使用偏导数而不是普通导数的原因。
Applying the Chain Rule to Our Neural Network / 将链式法则应用于我们的神经网络
Step 1: $\frac{\partial L}{\partial \hat{y}}$ 第一步：$\frac{\partial L}{\partial \hat{y}}$