CausalGate: Causal Importance Distillation for Transformer Module Pruning

CausalGate：用于 Transformer 模块剪枝的因果重要性蒸馏

Abstract: Existing adaptive inference methods for Large Language Models rely on observational heuristics, such as hidden-state similarity or activation magnitudes, to drop redundant modules. However, these correlation-based metrics often fail to capture subtle, non-linear structural computations vital for semantic accuracy.

摘要： 现有的针对大语言模型的自适应推理方法主要依赖于观测启发式算法（如隐藏状态相似度或激活幅度）来剔除冗余模块。然而，这些基于相关性的指标往往无法捕捉对语义准确性至关重要的细微、非线性结构计算。

We introduce CausalGate, an intervention-guided framework for compute-efficient transformer inference. During a calibration phase, CausalGate isolates individual Attention and MLP sub-layers, zeros out their respective outputs, and measures the exact semantic damage via the Kullback-Leibler divergence of the final logit distribution.

我们引入了 CausalGate，这是一个用于计算高效 Transformer 推理的干预引导框架。在校准阶段，CausalGate 会隔离各个 Attention 和 MLP 子层，将其输出置零，并通过最终 Logit 分布的 Kullback-Leibler 散度来衡量精确的语义损失。

To eliminate runtime routing overhead, this structural importance hierarchy is distilled into a global set of static, lightweight scalar gates using an Exponential Moving Average smoothing objective paired with a differentiable pairwise ranking loss.

为了消除运行时的路由开销，该结构重要性层级通过指数移动平均（EMA）平滑目标与可微成对排序损失相结合，被蒸馏为一组全局的静态轻量级标量门控（Scalar Gates）。

Evaluated on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B across language modeling and commonsense reasoning benchmarks, CausalGate consistently outperforms prominent dynamic routing and layer-skipping baselines, translating theoretical compute savings into concrete hardware latency reductions with zero operational overhead.

在 TinyLlama-1.1B、Qwen2.5-3B 和 Llama-3.1-8B 模型上，针对语言建模和常识推理基准的评估表明，CausalGate 在性能上始终优于主流的动态路由和层跳过（layer-skipping）基准方法，在零运行开销的情况下，将理论上的计算节省转化为具体的硬件延迟降低。