PANet Paper Walkthrough: When Feature Pyramids Go Bottom-Up

PANet 论文解读：当特征金字塔采用自底向上路径时

Deep Learning PANet Paper Walkthrough: When Feature Pyramids Go Bottom-Up Understanding how PANet shortens the path between low-level and high-level features. 深度学习 PANet 论文解读：当特征金字塔采用自底向上路径时——深入理解 PANet 如何缩短低层特征与高层特征之间的路径。

In my previous article I wrote about the FPN (Feature Pyramid Network) architecture [1], which is one of the most influential necks we can apply to a backbone model. FPN was first introduced to enhance the capability of an object detection model to detect small objects. However, Liu et al. in 2018 found that the information flow of FPN had room for improvement. So, they decided to address this gap by proposing PANet in their research paper titled “Path Aggregation Network for Instance Segmentation” [2], which we are going to discuss in this article. 在上一篇文章中，我介绍了 FPN（特征金字塔网络）架构 [1]，它是我们可以应用于主干网络（Backbone）中最具影响力的颈部（Neck）结构之一。FPN 最初被引入是为了增强目标检测模型对小目标的检测能力。然而，Liu 等人在 2018 年发现 FPN 的信息流仍有改进空间。因此，他们在题为《Path Aggregation Network for Instance Segmentation》（用于实例分割的路径聚合网络）的研究论文 [2] 中提出了 PANet，旨在解决这一问题，本文将对此进行探讨。

A Little Bit About FPN

关于 FPN 的简要回顾

PANet is built on top of the FPN architecture, so I think it would be a good idea to talk a little bit about how FPN works in advance. In a CNN-based backbone model, the deeper feature maps have different characteristics from the shallower ones. The feature maps from deeper layers tend to have high semantic information, yet it does not have that much amount of spatial information. Conversely, feature maps from shallower layers contain more spatial information but have less semantic information. These facts essentially tell us that we should use the deeper feature maps if we were to predict large objects and use the shallower feature maps for predicting small objects. PANet 是在 FPN 架构的基础上构建的，因此我认为有必要先简要介绍一下 FPN 的工作原理。在基于 CNN 的主干模型中，深层特征图与浅层特征图具有不同的特性。深层特征图往往包含丰富的语义信息，但空间信息较少；相反，浅层特征图包含更多的空间信息，但语义信息较少。这些事实告诉我们：如果我们要预测大目标，应该使用深层特征图；而预测小目标时，则应使用浅层特征图。

However, the authors of PANet found that it is not quite appropriate to directly take information from shallower feature maps to detect small objects. This is because these feature maps contain minimal amount of semantic information, which essentially implies that they do not have a good understanding about the content of the image. FPN solves this issue by combining feature maps from the deeper layers and the shallower ones. By doing so, we can basically inject semantic information into the shallower feature maps that they previously didn’t have. 然而，PANet 的作者发现，直接利用浅层特征图来检测小目标并不完全合适。这是因为这些特征图包含的语义信息极少，意味着它们对图像内容的理解不够深入。FPN 通过结合深层和浅层的特征图解决了这个问题。通过这种方式，我们可以将语义信息注入到原本缺乏这些信息的浅层特征图中。

What FPN Solves

FPN 解决了什么

Now take a look at the illustration in Figure 1 below to better understand this idea. The upward flow on the left is the main backbone model, while the one going downwards on the right (a.k.a. the top-down pathway) is the FPN. The horizontal arrows that go directly to the FPN part (called lateral connections) are the tensors to be combined with the one from the deeper layers. In this illustration, the FPN produces three feature maps where the detection head will make the bounding box predictions from. 现在请看下图 1，以更好地理解这一概念。左侧向上的流向是主干模型，而右侧向下的流向（即自顶向下路径）则是 FPN。直接指向 FPN 部分的水平箭头（称为横向连接）是将要与深层特征图进行融合的张量。在此图中，FPN 生成了三个特征图，检测头将基于这些特征图进行边界框预测。

(Figure 1. The FPN architecture [3].) （图 1. FPN 架构 [3]。）

The blue borderlines in the illustration above indicate the amount of semantic information contained within the corresponding feature map. You can see in the original backbone that the borderline gets thicker as we get deeper into the network, meaning that the feature map that has the highest semantic information is the one produced by the deepest layer. By using feature map combination mechanism proposed by FPN, we can see that the shallower feature maps are now rich of semantic information which makes them suitable for detecting small objects accurately. 上图中蓝色的边框表示相应特征图中所包含的语义信息量。你可以看到，在原始主干网络中，随着网络深度的增加，边框变得越厚，这意味着由最深层产生的特征图具有最高的语义信息。通过使用 FPN 提出的特征图融合机制，我们可以看到浅层特征图现在变得富含语义信息，这使得它们能够准确地检测小目标。

What FPN Doesn’t Solve

FPN 未解决的问题

At this point we might think that FPN is just fine as it is able to allow a model to detect small objects better compared to the plain backbone model. In fact, the authors of PANet still see something that isn’t solved by FPN just yet, which as I’ve mentioned earlier is related to the information flow problem. Now let’s take a look at Figure 2 below. 此时，我们可能会认为 FPN 已经足够好了，因为它确实比单纯的主干模型更能提升小目标的检测能力。事实上，PANet 的作者认为 FPN 仍有一些未解决的问题，正如我之前提到的，这与信息流问题有关。现在让我们看看下图 2。

(Figure 2. The PANet architecture [2].) （图 2. PANet 架构 [2]。）

In this figure, the component referred to as (a) is the backbone model that has been equipped with FPN, which is basically the same structure I showed you in Figure 1. In FPN, feature maps P₅, P₄, P₃, and P₂ become the basis of the detection head to make predictions, where P₂ has a high semantic information thanks to the flow in the top-down pathway. And here’s the problem FPN doesn’t solve: while FPN effectively enriches feature maps in shallower layers, the deeper feature maps still lack spatial information. 在此图中，标记为 (a) 的部分是配备了 FPN 的主干模型，这与我在图 1 中展示的结构基本相同。在 FPN 中，特征图 P₅、P₄、P₃ 和 P₂ 成为检测头进行预测的基础，其中 P₂ 得益于自顶向下的路径流，拥有了丰富的语义信息。这就是 FPN 未能解决的问题：虽然 FPN 有效地丰富了浅层特征图，但深层特征图仍然缺乏空间信息。

In the beginning of this article I said that the deeper and the shallower feature maps in a backbone have their own advantage and drawback, and FPN only solves the drawback of the shallower feature maps while still leaving the problem of the deeper feature maps untouched. Based on this notion, the authors of PANet thought that we could still improve the ability of an object detection model in detecting large objects by injecting spatial information to the deeper feature maps, which also potentially allows the overall accuracy of the detection model to get even higher. 在本文开头我提到过，主干网络中的深层和浅层特征图各有优缺点，而 FPN 只解决了浅层特征图的缺点，却对深层特征图的问题置之不理。基于这一理念，PANet 的作者认为，我们可以通过向深层特征图注入空间信息，进一步提升目标检测模型对大目标的检测能力，这也有望使检测模型的整体准确率得到进一步提高。

It is definitely normal for a deeper layer to lose spatial information due to the downsampling mechanism in the backbone. However, this is actually not solely caused by the downsampling itself. Instead, the long information travel in the stack of convolution layers also causes spatial information to degrade. So, if we cannot omit the downsampling operation within the backbone, we can still minimize this spatial information degradation by creating shortcut paths — just like the skip-connections in ResNet. 由于主干网络中的下采样机制，深层丢失空间信息是正常的。然而，这实际上不仅仅是由下采样本身引起的。卷积层堆叠中漫长的信息传递路径也会导致空间信息的退化。因此，如果我们无法省略主干网络中的下采样操作，我们仍然可以通过创建快捷路径（类似于 ResNet 中的跳跃连接）来最小化这种空间信息的退化。

How PANet Solves the Problem Unsolved by FPN

PANet 如何解决 FPN 未解决的问题

The authors of PANet proposed a clever idea, where they added a stack of several convolution layers placed on top of the existing FPN alongside the lateral connections (the part referred to as (b) in Figure 2). It might seem counterintuitive at glance — how can adding more layers allow us to get shorter paths instead? In fact, this approach is valid as it really allows the shallower feature maps to deliver spatial information to the deeper layers more seamlessly. Now let’s go back to Figure 2 and pay attention to the dashed arrows. Both arrows essentially denote the flow of the C₂ tensor (the backbone feature at the same spatial resolution as P₂ and N₂) to the deepest layer. The red one is when we only use the top-down pathway of FPN, while the green one is when… PANet 的作者提出了一个巧妙的想法：在现有 FPN 的基础上，沿着横向连接额外增加了一堆卷积层（即图 2 中标记为 (b) 的部分）。乍一看这似乎有悖常理——增加更多的层怎么反而能缩短路径呢？事实上，这种方法是有效的，因为它确实使浅层特征图能够更顺畅地将空间信息传递给深层。现在让我们回到图 2，注意虚线箭头。这两个箭头本质上都表示 C₂ 张量（与 P₂ 和 N₂ 具有相同空间分辨率的主干特征）流向最深层的路径。红色箭头表示仅使用 FPN 自顶向下路径的情况，而绿色箭头则表示……