Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem

Deleting whole transformer blocks is a cheap way to speed up a large language model: the model literally gets shorter, so block removal — depth pruning — buys predictable speedups and stacks with quantization. The hard part is which blocks to cut, because removing one block depends on which others go with it. The paper LLM Compression by Block Removal with Constrained Binary Optimization (Hugging Face, September 21, 2026) recasts that choice as a constrained binary optimization (CBO) mapping onto an Ising glass, whose energy is a cheap proxy for quality.

Why blocks are not independent

Existing methods score each block on its own — by magnitude, sensitivity or block influence — and remove the least important ones — a mean-field approximation that treats a block as independent of the others. Removing a single consecutive run of blocks is a common shortcut, but removing block 20 only hurts if blocks 19 or 24 went with it, and the number of combinations grows exponentially.

From block selection to energy minimization

Each block gets a binary variable: 0 keep, 1 remove. A second-order Taylor expansion of the model's loss gives an approximate Hessian: the diagonal captures each block's importance on its own, and the off-diagonal entries are the pairwise couplings mean-field methods discard. The task is to find the M blocks, out of N, whose removal minimizes the energy. The Hessian is computed once, from forward and backward passes on a small calibration dataset. Any candidate is then scored by a single cheap energy calculation, without running the model. The same Hessian serves many values of M, since the couplings do not depend on the compression target.

Solvers, spectrum and results

On a single GPU the authors brute-force tens of billions of configurations; the hardest tractable case, 8 of Llama-3.3-70B's 80 blocks (about 29 billion configurations), took roughly two days. Beyond that, the task in QUBO form goes to classical, quantum and quantum-inspired solvers — annealing, QAOA, tabu search, branch-and-bound — where an open-source tabu solver reaches the lowest-energy states in seconds. The true ground state is not required: a handful of good low-energy states suffices. For Llama-3.1-8B-Instruct at 16 of 32 blocks removed, the 17th excited state is the first to cut an early block and beats the ground state after light retraining. Without retraining, Llama-3.3-70B-Instruct reaches 76.6 and 76.9 MMLU at 32 and 40 of 80 blocks removed, against 59.3 and 54.0 for block influence; Qwen3-14B at 12 of 40 leads by about 10 points. It transfers to the hybrid NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 model too, beating block influence on AIME25 and GPQA. The code is open-sourced.