Data Density Scaling Laws for Image Self-Distillation

Current scaling law formulations suggest that increasing the number of unique samples in a dataset improves model performance, while repeated exposure to the same samples provides diminishing value.

目前的缩放定律公式表明，增加数据集中的唯一样本数量可以提高模型性能，而重复接触相同的样本则会带来边际效益递减。

In this work, we examine this assumption for image self-distillation models. Through systematic experiments across diverse datasets, namely Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M, we demonstrate that the effects of sample repetition on downstream performance are non-trivial and dataset-dependent.

在这项工作中，我们针对图像自蒸馏模型检验了这一假设。通过在 Maxar、SATLAS、Waymo、nuPlan、Ego4D、ImageNet 和 YFCC15M 等不同数据集上进行的系统性实验，我们证明了样本重复对下游性能的影响并非微不足道，且具有数据集依赖性。

We identify settings where repeatedly training on a smaller random subset can outperform training on a larger set of unique samples. This finding highlights a limitation of existing scaling laws.

我们确定了一些场景，在这些场景中，对较小的随机子集进行重复训练的效果可能优于对较大的唯一样本集进行训练。这一发现突显了现有缩放定律的局限性。

To address this gap, we propose a modified version of a Chinchilla-style scaling law that explicitly accounts for two factors: the number of unique samples and repetition per sample.

为了弥补这一差距，我们提出了一种改进版的 Chinchilla 风格缩放定律，该定律明确考虑了两个因素：唯一样本的数量和每个样本的重复次数。

This formulation provides a framework for understanding data-compute trade-offs and improves prediction in some regimes where repetition plays a central role, although its benefits are not consistent across datasets. Finally, we examine how subset selection and augmentation affect downstream performance.

该公式为理解数据与计算之间的权衡提供了一个框架，并在重复起核心作用的某些机制中改进了预测效果，尽管其益处在不同数据集之间并不一致。最后，我们研究了子集选择和增强如何影响下游性能。