FLOPs vs Real Work: The Importance of Replication in AI Efficiency Assessment

FLOPs 与实际工作量：复现研究在 AI 效率评估中的重要性

Abstract: AI efficiency has recently taken the spotlight in both academy and industry due to massive model scales, high energy demands, and environmental costs. While reporting Floating Point Operations (FLOPs) is a traditional approach for assessing computational costs, the relationship between FLOPs and execution time is not straightforward, as layers with the same number of FLOPs may not have the same execution time because some operations are more easily parallelized than others.

摘要： 由于模型规模的不断扩大、高昂的能源需求以及环境成本，AI 效率问题近期在学术界和工业界备受关注。虽然报告浮点运算次数（FLOPs）是评估计算成本的传统方法，但 FLOPs 与执行时间之间的关系并非直接对应；这是因为即使 FLOPs 数量相同的层，其执行时间也可能不同，原因在于某些操作比其他操作更容易实现并行化。

This paper sets out to replicate the original experiments from a study that proposed the $\alpha-FLOPs$ estimation formula to verify whether the results remain applicable on newer, more powerful hardware. During the replication process, we identify limitations in the replication materials provided by the original study, including a lack of specific dependency details and transparency regarding regression data.

本文旨在复现一项提出 $\alpha-FLOPs$ 估算公式的研究中的原始实验，以验证其结果在更新、更强大的硬件上是否依然适用。在复现过程中，我们发现了原始研究提供的复现材料存在局限性，包括缺乏具体的依赖项细节，以及在回归数据方面缺乏透明度。

Our results validate the thesis that raw FLOPs alone are not an appropriate metric for execution time, as spatial dimensions remain more easily parallelized than kernel dimensions. However, fine-grained measurements reveal that the relationship is much less straightforward than previously shown, with newer hardware exhibiting instabilities and discontinuities in execution time, including jumps and oscillations, that the $\alpha-FLOPs$ formula generally underestimates.

我们的研究结果证实了这样一个论点：仅凭原始 FLOPs 数值不足以作为衡量执行时间的指标，因为空间维度（spatial dimensions）依然比卷积核维度（kernel dimensions）更容易并行化。然而，细粒度的测量结果显示，两者之间的关系远比之前展示的要复杂得多。较新的硬件在执行时间上表现出不稳定性与不连续性，包括跳变和震荡，而 $\alpha-FLOPs$ 公式通常会低估这些现象。

Ultimately, this work validates the empirical findings from the original study but shows negative results when applying the $\alpha-FLOPs$ estimation. We also highlight the critical need for complete and accurate replication packages for research on hardware-dependent efficiency assessment and provide a complete replication package for our implementation to facilitate further study.

最终，本研究验证了原始研究的实证发现，但在应用 $\alpha-FLOPs$ 估算时得出了负面结果。我们还强调了在硬件依赖型效率评估研究中，提供完整且准确的复现包至关重要，并为我们的实现提供了完整的复现包，以促进后续研究。