HOPE: Higher-Order Pruning for Mixture-of-Experts Language Models

HOPE: Higher-Order Pruning for Mixture-of-Experts Language Models

Published November 9, 2026

Introduction

Mixture-of-Experts (MoE) models have made it possible to scale large language models to hundreds of billions of parameters. However, every expert multilayer perceptron (MLP) must remain in memory during inference, even though most experts are inactive for any individual token.

For example, Qwen3.5-122B contains 256 experts in each of its 48 MoE layers. Expert MLPs account for 95% of its approximately 250 GB parameter memory footprint, while only 8 experts per layer are used for a given token.

Expert pruning addresses this memory bottleneck by permanently removing the least useful experts. Pruning avoids the latency costs associated with dynamically loading and off-loading experts, requires no additional training, and can be combined with quantization. The main challenge is determining which experts can be removed safely.

Existing pruning methods score experts independently, assigning each one a scalar importance value based on a calibration dataset. They then remove the lowest-scoring experts independently within each layer. These are first-order methods because they treat each expert's contribution as independent.

In practice, experts often operate in cooperative groups. Experiments show consistent clusters of co-activated experts, while other research has found that expert cooperation is a stable property of MoE models and contributes to model performance. Ignoring these interactions can produce suboptimal pruning, particularly at aggressive pruning rates.

The paper Higher-Order Pruning of Experts in Mixture-of-Experts Language Models introduces HOPE, or Higher-Order Pruning of Experts. HOPE is a second-order method that accounts for pairwise interactions between experts. At a 50% pruning rate, it achieves the best average rank among the evaluated methods, with gains of up to 6.1% on agentic coding tasks over the next-best pruning method.

Experts do not work alone

In a standard MoE layer, there are E experts. Each input token x is routed independently to K experts, with K = 8 being a typical configuration. A gating layer produces softmax-normalized weights for the experts. The K experts with the highest gate weights form the active set T(x).

The layer output is:

sum over k in T(x) of g_k(x) f_k(x)

Here, g_k(x) is the gate weight and f_k(x) is the output of expert k for token x.

Experts in the active set are not independent. They form stable cooperative groups that are repeatedly selected together. Co-selection patterns can emerge early during training and remain stable afterward.

This matters when experts are pruned. Removing expert A may appear harmless when considered alone, but if expert B frequently cooperates with and relies on A, removing A can damage the entire cooperative group. First-order methods do not capture this effect because they score A and B separately.

The impact increases with the pruning rate. Removing up to 50% of the experts can break many cooperative links, which is where second-order pruning becomes most useful.

How HOPE identifies a prune set

When a set P of experts is removed from an MoE layer, the active expert set for a token may change. A pruned expert that would have been selected is replaced by the next-highest-weight available expert. The resulting change in the layer output is the pruning error.

The goal is to find a prune set of a predefined size that minimizes this error over tokens from a calibration dataset. Directly optimizing the error is intractable because the replacement experts depend on the prune set itself.

HOPE instead defines a tractable surrogate:

Z = [sum over j in P intersect T(x) of g_j(x) ||f_j(x)||_2]^2

This quantity strictly upper-bounds the dominant component of the true pruning error. Because it is a squared sum, expanding it produces cross-terms between pairs of experts:

g_i(x) g_j(x) ||f_i(x)||_2 ||f_j(x)||_2

These cross-terms represent pairwise interactions that first-order methods discard.

HOPE minimizes the expected value of Z over the calibration set. The problem can be expressed as a quadratic program:

P* = argmin_p p^T F p
subject to p in {0,1}^E
           sum over k from 1 to E of p_k = |P|

The interaction matrix F is defined as:

F_i,j = (1 / |X_i,j|) sum over x in X_i,j of
        g_i(x) g_j(x) ||f_i(x)||_2 ||f_j(x)||_2

where:

X_i,j = {x : i and j belong to T(x)}

The matrix is collected from the calibration dataset. HOPE then solves the quadratic program using a continuous relaxation before rounding the result to binary decisions.

Each diagonal entry F_k,k represents the individual importance of expert k. Off-diagonal entries F_i,j, where i differs from j, represent the pairwise cooperative contribution of experts i and j.

The structure of F varies by layer. In GLM-4.5-Air, which has 128 experts per layer and 106B total parameters, clustered visualizations show bright diagonal blocks at representative early, middle, and late layers. These blocks indicate groups of experts with strong pairwise interaction scores.

HOPE requires the same calibration pass as other expert-pruning methods: one forward pass over the calibration data. Its additional operation is solving the quadratic program. Although the problem is non-convex, a standard solver can process each layer in approximately 1 to 2 seconds. Calibration remains the dominant part of the process.

Connection to REAP

HOPE is closely related to Router-weighted Expert Activation Pruning (REAP), a first-order expert-pruning method. The diagonal of HOPE's F matrix encodes individual expert importance and effectively recovers the scores used by REAP, differing only because of the variance of expert contributions.

The diagonal of F recovers REAP's expert rankings with a Spearman correlation of ρ = 0.988. Therefore, improvements over REAP are attributable to the off-diagonal interaction terms in the F matrix.

Results

HOPE was evaluated on three models:

• Qwen3.5-122B
• Qwen3.5-35B
• GLM-4.5-Air

The evaluation used two calibration datasets, Evol-CodeAlpaca and SWE-Bench Verified trajectories. The resulting pruned models were tested with the Tulu3 suite and SWE-Bench Pro, an agentic coding benchmark.

HOPE was compared with four first-order baselines:

• REAP
• MAN
• EAN
• Frequency

Pruning rates ranged from 10% to 50%.

Across all models, calibration sets, benchmarks, and pruning rates, HOPE achieved the best average rank, 2.07 out of 5 methods. It produced the top result in 39% of conditions and ranked in the top two in 67% of conditions. In direct comparisons, HOPE outperformed each baseline in most conditions.

HOPE's advantage increased at higher pruning rates. At 50% pruning, its average rank improved to 1.58, compared with 2.42 for REAP, the next-best method. This setting also produces larger memory savings, which can support longer contexts, deeper reasoning, and more tool calls.

The largest advantage appeared on SWE-Bench Pro, where models autonomously solve real GitHub issues through multiple steps involving coding, tool calls, and reasoning. These tasks likely exercise diverse combinations of experts, making the preservation of cooperative structure particularly relevant.

At 50% pruning, Qwen3.5-35B-A3B achieved 42.3 on SWE-Bench Pro with HOPE, compared with 41.8 for the unpruned model. REAP achieved 36.2, a 13% regression relative to the unpruned model.

In some cases, pruned models matched or exceeded the unpruned baseline. Removing low-relevance experts may reduce noise on the target tasks, an effect also observed with other pruning methods. Code-focused calibration did not severely degrade non-coding benchmarks, and HOPE performed well across the Tulu3 suite regardless of the calibration set.

HOPE retains more cooperating experts

HOPE's advantage comes from optimizing over the full interaction matrix rather than only individual expert scores. To examine its practical effect, experts on which HOPE and REAP disagreed were analyzed using a cooperative score, defined as the average F value between a given expert and the other retained experts.

Experts retained only by HOPE consistently had higher cooperative scores than experts retained only by REAP. This shows that HOPE can sacrifice some individually high-scoring experts to preserve cooperative clusters. The benchmark results indicate that this trade-off improves performance.

Running HOPE

The HOPE package provides three steps:

1. Calibrate
2. Solve
3. Prune

It has been tested with Qwen3, Qwen3-Next, Qwen3.5, and GLM-4.5.

Step 1: Calibrate

hope calibrate \
    --model-path Qwen/Qwen3.5-35B-A3B \
    --prompts calibration_prompts.txt \
    --out-path observations.h5

Calibration runs one forward pass over the calibration data, which can consist of agentic rollouts or single-turn traces. Using Hugging Face transformers, the package hooks into the forward pass of each MoE layer.

For every token, it records the selected experts, gate weights, output norms, and pairwise co-activation statistics. The resulting F matrix is stored in an HDF5 file.

Step 2: Solve the quadratic program

hope solve \
    --obs-path observations.h5 \
    --prune-frac 0.50 \
    --out-path pruneset.json

This command materializes the F matrix and solves the binary quadratic program through a continuous relaxation. It takes a few seconds per layer and produces a JSON file containing the prune set.

Step 3: Prune the model

hope prune \
    --model-path Qwen/Qwen3.5-35B-A3B \
    --pruneset-path pruneset.json \
    --out-path ./Qwen3.5-35B-A3B-pruned-50

This command creates a smaller model checkpoint from the prune set. The resulting model is a standard Hugging Face checkpoint that is smaller on disk and in memory. No changes to vLLM or SGLang model definitions are required.

The process can also be called through a Python API:

from hope.calibrate import calibrate
from hope.solve import solve
from hope.prune import prune_model

calibrate("Qwen/Qwen3.5-35B-A3B", calibration_prompts, "observations.h5")
solve("observations.h5", prune_frac=0.50, out_path="pruneset.json")
prune_model("Qwen/Qwen3.5-35B-A3B", "pruneset.json", "./Qwen3.5-35B-A3B-pruned-50")

Limitations of HOPE

HOPE's advantage is most pronounced at pruning rates of 40% to 50%. At lower rates, such as 10% to 20%, first-order methods including REAP and MAN can sometimes be competitive.

Like most expert-pruning algorithms, HOPE prunes each layer independently and does not account for interactions across layers. A theoretical framework for cross-layer HOPE has been presented, but its results remain preliminary.

HOPE models pairwise interactions. Higher-order interactions involving triplets or larger groups could provide additional information, but estimating them is difficult because of their combinatorial complexity.

Further information

• Paper: Higher-Order Pruning of Experts in Mixture-of-Experts Language Models
• Code: HOPE on GitHub
• Hybrid architecture repository: Hybrid Model Factory

Citation

@article{tseng2026hope,
    title={Higher-Order Pruning of Experts in Mixture-of-Experts Language Models},
    author={Tseng, Alex M. and Kaul, Prannay and Zancato, Luca and Xia, Wei and Soatto, Stefano},
    journal={arXiv preprint arXiv:2609.18916},
    year={2026},
}