Neural Networks: Weights, Activation, and Backpropagation

神经网络：权重、激活函数与反向传播

If you’ve ever looked at a neural network diagram and ever wondered what is actually happening in these networks, this post is just for you. I pulled this post together after researching the properties of neural networks, and writing out what is finally there to click. No analogies that lead to no explanations, just the math behind neural networks, one piece at a time. 如果你曾看过神经网络图并好奇这些网络内部究竟在发生什么，那么这篇文章正是为你准备的。在研究了神经网络的特性并梳理出核心逻辑后，我整理了这篇文章。这里没有那些无法解释问题的类比，只有神经网络背后的数学原理，我们将逐一拆解。

The Simplest Possible Net: A Perceptron

最简单的网络：感知机 (Perceptron)

Before neural networks got deep, they started shallow. A perceptron is a neural net with zero hidden layers, invented back in 1943. Perceptrons work when data is linearly separable, meaning you could draw a line and cleanly split the classes on either side of it. Let’s take a simple example with an equation: D=y-2x-3. Everything where D>=0 is Class 1 and everything where D<0 is Class 2. This is basically how a model works: a single linear boundary that decides which side the example falls on. 在神经网络变得“深”之前，它们始于“浅”。感知机是一种没有隐藏层的神经网络，发明于 1943 年。当数据是线性可分时，感知机就能发挥作用，这意味着你可以画一条线，将不同类别清晰地分开。让我们看一个简单的方程例子：D=y-2x-3。所有 D>=0 的点属于类别 1，所有 D<0 的点属于类别 2。这基本上就是模型的工作方式：通过单一的线性边界来决定样本落在哪个区域。

However, real data is rarely this cooperative. The moment your classes tangle in a way no straight line can separate, you will need more than one layer. This is really where the deep learning begins. 然而，现实数据很少如此配合。一旦你的类别交织在一起，无法用直线分开时，你就需要不止一层。这正是深度学习的起点。

What A Neuron is Actually Computing

神经元究竟在计算什么

Zoom into any single neuron in a network, and this is the entire computation occurring inside of it: S = b + ∑ (w_i * x_i). 放大观察网络中的任何单个神经元，其内部发生的全部计算如下：S = b + ∑ (w_i * x_i)。

x_i are the inputs coming in, such as the pixel values and whatever your data is. w_i are the weights (how much the network currently believes each input matters). b is the bias that acts like a nudge that shifts the output independent of the inputs. S is the logit, which is an unbounded value. The S value either shoots in the negative or positive direction indefinitely. x_i 是输入值，例如像素值或任何你的数据。w_i 是权重（网络当前认为每个输入的重要性程度）。b 是偏置，像是一个微调项，在不依赖输入的情况下改变输出。S 是 Logit，这是一个无界值，可以无限向负方向或正方向延伸。

Stack raw logits like this across layers with nothing else added, and the whole network has a mathematical collapse into a linear function. Depth alone buys you nothing. You need a twist. 如果只是将这些原始 Logit 在层与层之间堆叠而不做其他处理，整个网络在数学上会坍缩成一个线性函数。仅靠深度本身毫无意义，你需要一个“转折”。

The Non-Linear Twist That Makes Depth Matter

让深度产生意义的非线性转折

That very twist is the activation function. This is applied to every logit before it moves onto the next layer. One of them is sigmoid: ø(S) = 1 / (1 + e^-S). 这个转折就是激活函数。它在 Logit 进入下一层之前被应用。其中一种是 Sigmoid 函数：ø(S) = 1 / (1 + e^-S)。

Just as the equation says, the ø(S) pushes the S value closest to 1 as the S value is positive. If S is negative, then the ø(S) value moves towards 0. This nonlinearity is the backbone of how depth works. With nonlinearity, each added layer can fold the decision boundary into complex shapes, which is exactly what is required to separate tangled data. 正如方程所示，当 S 为正时，ø(S) 将值推向 1；如果 S 为负，则 ø(S) 的值趋向于 0。这种非线性是深度学习运作的基石。有了非线性，每一层增加的层数都能将决策边界折叠成复杂的形状，这正是分离交织数据所必需的。

Stacking Neurons into Layers

将神经元堆叠成层

A layer is just a bank of neurons, each computing its own weighted sum from the same inputs, and each with its own weights. The formula goes like this: z_i = ϕ (∑ x_j * w_ij). 层只是一组神经元，每个神经元根据相同的输入计算自己的加权和，并拥有各自的权重。公式如下：z_i = ϕ (∑ x_j * w_ij)。

The middle layers are called the hidden layers. They are called hidden layers because the values are not directly observed; only the input and the output (final value) are observed. The early layers tend to pick up the simple patterns while the deeper layers form abstract connections with the simple patterns made by the early layers. 中间的层被称为隐藏层。之所以叫隐藏层，是因为这些值无法被直接观察到；只有输入和输出（最终值）是可见的。早期的层倾向于捕捉简单的模式，而更深的层则基于早期层形成的简单模式构建抽象连接。

The final output layer turns those hidden features into a prediction: y_i = ϕ(∑ z_j * w_ij). For a K-way classification, the K output neurons are used. These neurons are used as one confidence score per class. 最终的输出层将这些隐藏特征转化为预测：y_i = ϕ(∑ z_j * w_ij)。对于 K 分类问题，会使用 K 个输出神经元。这些神经元被用作每个类别的置信度分数。

Softmax: Turning Raw Scores into Real Probabilities

Softmax：将原始分数转化为真实概率

Say that the output layer displays three logits: (0.01, -3.8, 4.2). These numbers are all currently unbounded and do not hold any significant meaning at the moment. Softmax fixes this by taking any of those three logits and converting them into probabilities between 0 and 1 that sum up to exactly 1.0. The formula for softmax is: p_j = (e^z_j) / (∑ e^z_k). 假设输出层显示三个 Logit：(0.01, -3.8, 4.2)。这些数字目前都是无界的，暂时没有明确的意义。Softmax 通过将这三个 Logit 转换为 0 到 1 之间且总和恰好为 1.0 的概率来解决这个问题。Softmax 的公式为：p_j = (e^z_j) / (∑ e^z_k)。

In the example above, the 4.2 value would come out closer to a 90% or higher confidence score, while -3.8 will approach closer to 0, a lower confidence score. 在上面的例子中，4.2 这个值会转化为接近 90% 或更高的置信度分数，而 -3.8 则会趋近于 0，即较低的置信度分数。

Loss Functions: Scoring the Damage

损失函数：评估误差

Once the network makes a prediction, there must be a way to measure how inaccurate it was. For numerical predictions, it is normally considered the Mean Squared Error. For classification, it is almost always Cross-Entropy. This is how the formula goes: CE = -∑ [y_i * log(p(y_i)) + (1-y_i) * log(1-p(y_i))]. 一旦网络做出预测，就必须有一种方法来衡量它的不准确程度。对于数值预测，通常使用均方误差 (MSE)。对于分类问题，几乎总是使用交叉熵 (Cross-Entropy)。公式如下：CE = -∑ [y_i * log(p(y_i)) + (1-y_i) * log(1-p(y_i))]。

The intuition is quite simple: if the true label is 1 and the model confidently predicts closer to 0, the log term blows up and there are severe penalties. However, if the model was correct, the penalty shrinks towards zero. Cross-entropy rewards confident correctness and punishes the confident mistakes. This is exactly what you want out of a loss function. 其直觉非常简单：如果真实标签是 1，而模型自信地预测接近 0，对数项会变得非常大，从而产生严重的惩罚。然而，如果模型预测正确，惩罚会趋向于零。交叉熵奖励自信的正确，惩罚自信的错误。这正是你对损失函数的要求。

What the Network Actually Learns: Backprop and Gradient Descent

网络究竟学到了什么：反向传播与梯度下降

Here’s the loop that turns a “useless” network into a “useful” network: 以下是将“无用”网络转化为“有用”网络的循环：

1. Feed a batch of training samples through the network. (将一批训练样本输入网络。)
2. Compute the predictions as the output layer. (在输出层计算预测结果。)
3. Measure losses and how inaccurate the predictions were. (衡量损失以及预测的不准确程度。)
4. Backpropagate the error backwards through every layer so you can calculate the weight contributed to the error. (将误差反向传播到每一层，以计算每个权重对误差的贡献。)
5. Nudge every weight slightly in the direction that reduces the error. (向减少误差的方向微调每个权重。)

That fifth step is called the gradient descent: (w_ij)^new = w_ij - α * (∂E / ∂w_ij). α holds the learning rate. Just picture the loss as a landscape. Training is descending that landscape, one gradient-calculated step at a time, trying to reach a low point. In a network with millions of billions of weights, that landscape has many dimensions with multiple valleys. This is part of why training is not guaranteed to find the global best solution, just a good one. 第五步被称为梯度下降：(w_ij)^new = w_ij - α * (∂E / ∂w_ij)。α 代表学习率。试着把损失想象成一片地形。训练过程就是在这片地形上，通过梯度计算一步步向下走，试图到达低谷。在一个拥有数百万甚至数十亿权重的网络中，这片地形拥有许多维度和多个山谷。这就是为什么训练不能保证找到全局最优解，只能找到一个较优解的原因之一。

One pass through the entire training set is called an epoch. Networks are trained across many epochs because each pass leaves weights in a slightly different place. This is why seeing the same data again isn’t wasted effort. 遍历整个训练集一次称为一个 Epoch。网络需要经过多个 Epoch 的训练，因为每一次遍历都会使权重处于略微不同的位置。这就是为什么重复看到相同的数据并非浪费时间。