Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
像物理学家一样剪枝大模型：将层移除视为伊辛模型优化问题
One of the cheapest ways to make a large language model faster is also one of the bluntest: delete whole transformer blocks. Because the model literally gets shorter, block removal (also called depth pruning) buys predictable inference speedups on top of the memory savings, and it stacks cleanly with quantization, low-rank compression, and other techniques. 让大语言模型提速最廉价的方法之一，也是最简单粗暴的方法：直接删除整个 Transformer 层。由于模型在物理层面变短了，层移除（也称为深度剪枝）不仅能节省内存，还能带来可预测的推理加速，并且可以与量化、低秩压缩等其他技术完美叠加。
The hard part is deciding which blocks to cut. Remove the wrong ones and the model collapses; and the effect of removing any one block depends on which others you remove alongside it, so the choices interact. That makes it a combinatorial problem, not a ranking problem, and combinatorial problems with interacting binary variables are exactly what the physics of spin systems was built to describe. 难点在于决定剪掉哪些层。如果剪错了，模型性能就会崩溃；而且移除某一层的影响取决于你同时移除了哪些其他层，因此这些选择之间存在相互作用。这使得它成为一个组合优化问题，而非简单的排序问题。而带有相互作用二元变量的组合问题，正是物理学中自旋系统理论所要描述的对象。
Our latest paper, LLM Compression by Block Removal with Constrained Binary Optimization, takes that correspondence literally. We reformulate block selection as a constrained binary optimization (CBO) problem that maps directly onto an Ising glass, a disordered spin system with all-to-all interactions and a fixed number of “up” spins. 我们最新的论文《通过约束二元优化进行层移除的大模型压缩》正是利用了这种对应关系。我们将层选择重新表述为一个约束二元优化（CBO）问题，并将其直接映射到伊辛玻璃（Ising glass）模型上——这是一种具有全连接相互作用且“向上”自旋数固定的无序自旋系统。
The energy of that spin system turns out to be a strong, cheap proxy for how well the pruned model will actually score on benchmarks, which means we can rank a huge number of candidate configurations without benchmarking any of them, and hand the hard instances to the same classical and quantum-inspired solvers we use elsewhere at Multiverse. 事实证明，该自旋系统的能量是衡量剪枝后模型在基准测试中表现的一个强大且廉价的代理指标。这意味着我们无需对任何候选配置进行实际测试，就能对海量配置进行排序，并将复杂的实例交给我们在 Multiverse 其他项目中使用的经典求解器或量子启发式求解器来处理。
The payoff in the deep-compression regime is large: at 50% compression of Llama-3.3-70B-Instruct, we gain almost 23 percentage points on MMLU over the best competing block-removal method. 在深度压缩场景下，这种方法收益巨大：在对 Llama-3.3-70B-Instruct 进行 50% 的压缩时，我们的 MMLU 得分比目前最优秀的同类层移除方法高出近 23 个百分点。
Why picking blocks is a many-body problem
为什么选择层是一个多体问题
Most existing block-removal methods score each block on its own, then remove the ones that look least important, using magnitude, sensitivity, or “block influence” heuristics. In physics terms these are mean-field methods: they treat each block as if its contribution were independent of the others, the way mean-field theory replaces a spin’s neighbors with a single averaged field. 目前大多数层移除方法都是单独评估每一层，然后利用幅度、敏感度或“层影响力”等启发式方法移除看起来最不重要的层。用物理学术语来说，这些是平均场方法：它们将每一层视为独立于其他层，就像平均场理论用单一的平均场来代替自旋的邻居一样。
A related shortcut is to only ever remove a single consecutive run of blocks, which keeps the problem small but throws away most of the search space. The trouble is that blocks are not independent, any more than spins in a real magnet are. Whether removing block 20 hurts the model depends on whether you also removed block 19 or block 24, an interaction, or coupling, between the two decisions. 另一种相关的捷径是只移除连续的一段层，这虽然简化了问题，却丢弃了大部分搜索空间。问题在于，层与层之间并非独立，就像真实磁体中的自旋并非独立一样。移除第 20 层是否会损害模型，取决于你是否同时移除了第 19 层或第 24 层——这就是两个决策之间的相互作用或耦合。
As models get deeper and more heterogeneous, ignoring those couplings leaves quality on the table, especially when you want to remove a lot of blocks at once. What you really want is to search over combinations of blocks while accounting for how they interact, but the number of combinations grows exponentially, so brute force looks hopeless. This is precisely the regime, exponentially large configuration spaces with pairwise couplings, where the tools of statistical physics earn their keep. 随着模型变得越来越深、越来越异构，忽略这些耦合会导致性能损失，尤其是在你想一次性移除大量层时。你真正想要的是在考虑层间相互作用的同时搜索层组合，但组合数量呈指数级增长，暴力搜索显得毫无希望。这正是统计物理工具大显身手的领域：处理具有成对耦合的指数级庞大配置空间。
The idea: turn block selection into an energy-minimization problem
核心思路：将层选择转化为能量最小化问题
We attach a binary variable to each transformer block: 0 means keep it, 1 means remove it, just like a spin that can point down or up. Then we do a second-order Taylor expansion of the model’s loss with respect to those variables, which produces an (approximate) Hessian matrix. The diagonal of that Hessian is how much each block matters on its own; the off-diagonal entries are exactly the pairwise couplings between blocks, the many-body physics that mean-field methods throw away. 我们为每个 Transformer 层分配一个二元变量：0 表示保留，1 表示移除，就像指向向下或向上的自旋一样。然后，我们对模型损失函数关于这些变量进行二阶泰勒展开，从而得到一个（近似的）海森矩阵。海森矩阵的对角线元素代表每一层自身的重要性；而非对角线元素正是层与层之间的成对耦合，这正是平均场方法所丢弃的多体物理信息。
That reformulation turns “which blocks should I remove?” into a clean optimization: find the set of M blocks whose removal minimizes the energy xᵀH⁰x, subject to removing exactly M of the N blocks. Mathematically this is a constrained binary optimization problem; physically it is an Ising glass, an all-to-all coupled spin system with conserved magnetization (the fixed number of removed blocks plays the role of a fixed total spin). 这种重构将“我应该移除哪些层？”的问题转化为一个清晰的优化问题：在必须从 N 层中移除 M 层的前提下，找到一组能使能量 xᵀH⁰x 最小化的 M 层组合。在数学上，这是一个约束二元优化问题；在物理上，它是一个伊辛玻璃模型，即一个具有守恒磁化强度的全连接自旋系统（移除层的固定数量充当了固定总自旋的角色）。
The key property we establish is that this energy is a strong proxy for downstream quality: low-energy states of the spin system correspond to high-performing pruned models. Minimizing energy and maximizing benchmark score become the same search. 我们确立的关键特性是：该能量是下游模型质量的强代理指标——自旋系统的低能态对应于高性能的剪枝模型。最小化能量与最大化基准测试得分变成了同一个搜索过程。
The reason this is practical is cost. The Hessian, i.e. the full set of couplings, is computed just once, from forward and backward passes on a small calibration dataset. After that, evaluating any candidate configuration is a single cheap energy calculation, no need to run the actual model, let alone benchmark it. And because the couplings don’t depend on the compression target, the same Hessian can be reused to solve for many different values of M. 这种方法之所以实用，是因为成本低廉。海森矩阵（即全套耦合参数）只需在少量校准数据集上通过一次前向和反向传播计算即可。此后，评估任何候选配置只需进行一次廉价的能量计算，无需运行实际模型，更无需进行基准测试。而且由于耦合参数不依赖于压缩目标，同一个海森矩阵可以重复用于求解不同的 M 值。
Solving it: exact when you can, quantum or quantum-inspired when you can’t
求解：能精确求解时精确求解，不能时使用量子或量子启发式方法
For most models the configuration space is large but still checkable. Because computing one energy is so cheap, we brute-force it on a single GPU, checking up to tens of billions of spin configurations. A few million take seconds; the hardest tractable case here, removing 8 of Llama-3.3-70B’s 80 blocks (about 29 billion configurations), took roughly two days. Beyond that the exact approach breaks down, and this is where casting the problem as an Ising glass pays off a second time. In its equivalent QUBO form (the constraint absorbed into a penalty term), the exact same task can be handed to the highly optimized classical, 对于大多数模型，配置空间虽然巨大但仍可检查。由于计算一次能量非常廉价，我们在单块 GPU 上进行暴力搜索，检查多达数百亿种自旋配置。几百万种配置只需几秒钟；这里最难的可处理案例是移除 Llama-3.3-70B 80 层中的 8 层（约 290 亿种配置），耗时约两天。在此之外，精确方法就会失效，而这正是将问题转化为伊辛玻璃模型的第二次收益所在。在其等效的 QUBO（二次无约束二元优化）形式下（约束被吸收到惩罚项中），同样的任务可以交给高度优化的经典求解器……