Optimal Pruning for Neural Architectures using Fisher Information Distances

基于费舍尔信息距离的神经网络架构最优剪枝

Abstract: A new scheme for parameter pruning is introduced, derived from the differential-geometric distance in model space. Pruning a parameter sets its value to zero, representing a displacement of the model to the hypersurface on which that parameter vanishes. The minimal distance from the unpruned model to this hypersurface is naturally computed via the geodesic distance in the model space as determined by the Fisher information metric. This distance determines the true change in the model, and its performance, under pruning.

摘要： 本文引入了一种新的参数剪枝方案，该方案源于模型空间中的微分几何距离。剪枝一个参数意味着将其值设为零，这代表了模型向该参数消失的超曲面发生位移。从未剪枝模型到该超曲面的最小距离，可以通过由费舍尔信息度量（Fisher information metric）确定的模型空间中的测地线距离自然地计算出来。该距离决定了模型在剪枝下的真实变化及其性能表现。

By analysing progressively more faithful approximations of this geodesic distance a natural hierarchy of optimality for pruning methods is determined. This starts with the traditional magnitude pruning, then develops into new more sophisticated and effective pruning schemes. The method is demonstrated for both fully-connected networks and vision transformers, on MNIST and CIFAR-10, over the complete 0-100% pruning range and across five random seeds.

通过分析对该测地线距离越来越精确的近似，确定了一种剪枝方法的最优性自然层级。该层级从传统的幅度剪枝开始，进而发展出更复杂且有效的剪枝方案。该方法在全连接网络和视觉 Transformer 上进行了验证，数据集涵盖 MNIST 和 CIFAR-10，剪枝范围覆盖 0-100%，并跨越了五个随机种子。

It outperforms pruning by parameter magnitude and by the local Fisher information alone in every architecture and dataset combination considered, on both accuracy and the Matthews correlation coefficient. Additionally, analysis of different levels of geodesic approximation produces intermediate pruning schemes that are computationally efficient and maintain near-optimal performance. This geometric picture supplies not only a state-of-the-art pruning methodology for AI models, but also a verified and mathematically-motivated justification for pruning schemes.

在所考虑的每一种架构和数据集组合中，该方法在准确率和马修斯相关系数（Matthews correlation coefficient）方面均优于仅基于参数幅度和局部费舍尔信息的剪枝方法。此外，对不同测地线近似水平的分析产生了一些中间剪枝方案，这些方案在计算上高效且保持了近乎最优的性能。这种几何图景不仅为人工智能模型提供了一种最先进的剪枝方法论，还为剪枝方案提供了一种经过验证且具有数学动机的理论依据。