Pruning LLMs as an Ising Optimization Problem

Pruning LLMs as an Ising Optimization Problem

Removing entire transformer blocks is one of the simplest ways to make a large language model faster. Because the model becomes physically shorter, block removal, also known as depth pruning, can provide predictable inference speedups in addition to reducing memory use. It can also be combined with quantization, low-rank compression, and other optimization techniques.

The difficult question is deciding which blocks to remove. Deleting the wrong blocks can severely damage model quality, and the effect of removing one block depends on which other blocks are removed. This makes block selection a combinatorial optimization problem rather than a simple ranking task. Combinatorial problems with interacting binary variables are closely related to the physics of spin systems.

The paper LLM Compression by Block Removal with Constrained Binary Optimization formulates block selection as a constrained binary optimization (CBO) problem. The resulting system maps to an Ising glass, a disordered spin system with all-to-all interactions and a fixed number of up spins. Its energy provides a useful, inexpensive proxy for the performance of a pruned model, allowing many candidate configurations to be ranked without benchmarking each one.

At 50% compression of Llama-3.3-70B-Instruct, the method improves MMLU by almost 23 percentage points compared with the strongest competing block-removal method.

Why block selection is a many-body problem

Many block-removal methods evaluate blocks individually and remove those considered least important. They may use magnitude, sensitivity, or block-influence heuristics. In physics terms, these are mean-field approaches: they treat each block as though its contribution were independent of the others.

Another shortcut is to remove only one consecutive sequence of blocks. This reduces the search space, but it excludes most possible configurations.

In practice, blocks are not independent. Removing block 20 may have a different effect depending on whether blocks 19 or 24 are also removed. These interactions, or couplings, become increasingly important as models grow deeper and more heterogeneous, particularly when many blocks must be removed simultaneously.

Searching all combinations is exponentially expensive. The problem therefore fits the setting in which statistical-physics methods are useful: large configuration spaces with pairwise interactions.

Converting block selection into energy minimization

A binary variable is assigned to every transformer block:

• 0 means the block is kept.
• 1 means the block is removed.

The method then uses a second-order Taylor expansion of the model's loss with respect to these variables. This produces an approximate Hessian matrix. The diagonal entries describe the individual importance of each block, while the off-diagonal entries represent pairwise couplings between blocks.

The optimization task becomes finding the set of M blocks whose removal minimizes the energy xᵀH⁰x, subject to removing exactly M of the model's N blocks. Mathematically, this is a constrained binary optimization problem. Physically, it corresponds to an Ising glass with a fixed magnetization, because the number of removed blocks is fixed.

The central observation is that low-energy configurations tend to produce high-performing pruned models. Minimizing the energy therefore provides a way to search for configurations that maximize downstream benchmark performance.

The method's computational cost is limited because the Hessian is calculated once using forward and backward passes over a small calibration dataset. Afterward, evaluating a candidate configuration requires only a low-cost energy calculation. The model does not need to run for every candidate, and the candidates do not need to be benchmarked individually. Since the couplings do not depend on the compression target, the same Hessian can be reused for different values of M.

Exact and heuristic solution methods

For some models, the configuration space is large but still small enough to enumerate. Since each energy evaluation is inexpensive, the method can brute-force tens of billions of configurations on a single GPU. A few million configurations take seconds. The hardest tractable case reported in the study involved removing 8 of the 80 blocks in Llama-3.3-70B, or approximately 29 billion configurations, and took about two days.

Beyond that scale, exhaustive search becomes impractical. The Ising formulation can instead be converted into an equivalent QUBO formulation, with the block-count constraint represented by a penalty term. The resulting problem can be submitted to classical, quantum, and quantum-inspired solvers, including methods based on quantum annealing, QAOA, tabu search, and specialized branch-and-bound.

An open-source tabu solver reached the lowest-energy states in seconds, including on difficult cases that could be checked against brute-force results. The objective is not necessarily to find the exact ground state. It is sufficient to generate several strong low-energy configurations, which is a less demanding task and allows lightweight solvers to be run multiple times.

Why the low-energy spectrum matters

Energy is a strong but imperfect proxy for model quality. As a result, the single lowest-energy configuration is not always the best model. The optimization setup also provides access to low-lying excited states, producing a group of high-quality pruning candidates rather than one potentially fragile answer.

For Llama-3.1-8B-Instruct with 16 of 32 blocks removed, most of the highest-ranked states removed blocks near the end of the model. However, the 17th excited state was the first to remove a block near the beginning. After light retraining, that configuration outperformed the ground state on several benchmarks.

This result challenges the assumption that effective pruning must remove one consecutive group of middle or late blocks. It also illustrates the value of considering the full set of block interactions.

Results

The method was evaluated on Llama-3.1-8B-Instruct, Qwen3-14B, and Llama-3.3-70B-Instruct. It performed at least as well as, and often better than, existing block-removal baselines. The difference increased as compression became more aggressive.

For Llama-3.3-70B-Instruct without retraining, CBO was approximately comparable to block influence through the removal of 24 of 80 blocks. At 32 and 40 removed blocks, it performed substantially better. At the deepest compression level, its MMLU score was almost 23 points higher than the baseline, and it outperformed the baseline on every tested benchmark.

For Qwen3-14B with 12 of 40 blocks removed, CBO led block influence on MMLU by approximately 10 points. At lighter compression levels, the methods were more similar, consistent with the idea that block couplings become more important when more blocks are removed.

Llama-3.3-70B-Instruct, no retraining

At 40 of 80 blocks removed, CBO kept MMLU close to 77, while the strongest baseline fell to the mid-50s.

Application to heterogeneous models

Block removal is more complicated in modern architectures that interleave different types of layers. The Ising formulation does not require every site to contain the same kind of block. A coupling can be represented regardless of whether the corresponding site contains an attention, Mamba2, or mixture-of-experts layer.

The method was tested on NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, a hybrid model that combines Mamba2, attention, and mixture-of-experts layers in a non-uniform arrangement. No retraining was used.

When removing two or three MoE layers or two attention layers, CBO found configurations that outperformed block influence on AIME25 and GPQA. The results also indicated that redundancy is unevenly distributed in these hybrid models. Some expert layers were more disposable than others, and searching the coupled configuration space helped identify effective removal patterns.

As with the dense models, the best configuration was often an excited state rather than the ground state.

Combining block removal with other compression methods

The CBO formulation can be used alongside quantization, low-rank and SVD compression, width pruning, and knowledge-distillation-based recovery. Block removal can therefore serve as one stage in a broader model-compression pipeline.

The approach also provides a framework for applying classical, quantum, and quantum-inspired optimization methods to the problem of selecting blocks in large language models.