Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learning

Masked Swingers：利用数据增强推进自监督学习中的自动编码器

Abstract: Self-supervised learning (SSL) removes the need for annotations and makes models that are capable across more domains than supervised learning. The autoencoder SSL framework learns by reconstructing its own input after information loss through a bottleneck or noise injection. Masked autoencoders (MAE) are the most successful instantiation of this framework: they encode a random subset of patches, then decode the masked-out patches.

摘要： 自监督学习（SSL）消除了对标注的需求，并使模型能够比监督学习跨越更多的领域。自动编码器 SSL 框架通过在经过瓶颈层或噪声注入导致信息丢失后，重建其自身输入来进行学习。掩码自动编码器（MAE）是该框架最成功的实例：它们对随机子集的图像块进行编码，然后解码被掩盖的图像块。

In this work, we introduce key modifications to improve MAEs. Our method augments an image in two different ways, then masks and encodes each view separately. It then exchanges the global representations (CLS tokens) between views before decoding the masked patches. By design, our Masked Swingers encourages learning a view-agnostic summary of the image to facilitate efficient transfer.

在这项工作中，我们引入了关键的改进来优化 MAE。我们的方法以两种不同的方式增强图像，然后分别对每个视图进行掩码和编码。在解码被掩盖的图像块之前，它会在视图之间交换全局表示（CLS token）。通过这种设计，我们的 Masked Swingers 鼓励学习图像的“视图无关”摘要，从而促进高效的迁移学习。

We perform extensive experiments, and find Masked Swingers outperforms MAE by +3-5% on ImageNet-1K kNN and provides large gains on fine-grained tasks, e.g., relative gains of +45% on instance retrieval, +22% on animal re-ID, and +76% on Omniglot character recognition. To boot, Swingers reduces error -64% relative to MAE on three new state-probing datasets, opening the door to world modeling. Welcome to our Swingers party.

我们进行了广泛的实验，发现 Masked Swingers 在 ImageNet-1K kNN 上比 MAE 提升了 +3-5%，并在细粒度任务上提供了巨大的增益，例如：实例检索相对提升 +45%，动物重识别（re-ID）提升 +22%，Omniglot 字符识别提升 +76%。此外，Swingers 在三个新的状态探测数据集上将错误率较 MAE 降低了 -64%，为世界建模打开了大门。欢迎来到我们的 Swingers 派对。