REAP Explained: How Cerebras Prunes MoE Experts to Shrink GLM and Qwen Models

A modern Mixture-of-Experts (MoE) model has a strange property: it is cheap to run but expensive to own. A model like GLM-4.6 activates only about 32 billion of its 355 billion parameters for any given token, so the math per token is modest. But every one of those 355 billion parameters still has to sit in GPU memory, waiting its turn. That memory bill, not the compute, is what keeps the best open MoE models off the hardware most people actually have.

REAP is a direct answer to that problem. Short for Router-weighted Expert Activation Pruning, it is a method from Cerebras that permanently deletes the least useful experts from a trained MoE model in a single pass, with no retraining. The result is a smaller model you can download today. Cerebras has already used it to publish compressed versions of GLM-4.5-Air, GLM-4.6, GLM-4.7, Qwen3-Coder, MiniMax-M2, Kimi-Linear, and more on Hugging Face, some of them 40 percent lighter while scoring within a point or two of the original on coding benchmarks.

This article walks through why MoE models waste memory, the two ways to reclaim it, why the obvious approach (merging experts) quietly hurts quality, and the surprisingly simple saliency score that lets REAP prune instead. Then we look at what the numbers actually show and which checkpoints you can run.

Here is the whole method in four steps. Step through it now; the rest of the article explains why each step is the right call.

The problem: sparse compute, dense memory

To see why pruning matters, it helps to separate two costs that MoE models deliberately pull apart.

The first is compute per token. A dense 70B model runs all 70 billion parameters for every token. An MoE layer instead keeps many parallel "expert" feed-forward networks and a small router that sends each token to only a handful of them, usually the top 8 out of a hundred or more. Only the selected experts run, so the compute stays close to that of a much smaller dense model. That is the whole point of the design, and it is why we now have open models with hundreds of billions of parameters that are still practical to serve.

The second cost is memory. Sparse routing decides which experts run, but it does not decide which experts must be present. Any token might route to any expert, so all of them have to be loaded. In GLM-4.6, the 160 routed experts per layer dominate the parameter count. You pay for 355 billion parameters of storage to use roughly 32 billion of them at a time. On a single 80 GB accelerator, that gap is the difference between "runs comfortably" and "does not fit."

Two facts make this gap attackable. Expert utilization is uneven: the router leans on some experts far more than others. And some experts contribute little even when they are chosen, producing small output vectors that barely move the layer's result. If you could identify the experts that are both rarely used and low-impact, you could remove them and shrink the memory bill without touching the parts of the model that do the work. That is exactly the opening REAP targets.

Two ways to shrink an MoE: merge or prune

Once you accept that some experts are expendable, there are two families of methods for acting on that.

Merging combines several experts into one. If experts 3, 7, and 12 look similar, replace them with a single averaged expert and rewire the router to point at it. The appeal is that you keep some information from every expert rather than throwing any away. Merging has been the more fashionable direction in the MoE compression literature, with methods such as M-SMoE and HC-SMoE.

Pruning simply deletes experts. Pick the least useful ones, remove their parameters, and drop their entries from the router. It feels more destructive, because you are discarding capacity outright.

The intuition that merging is gentler than pruning is exactly what the REAP paper, titled "REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression," sets out to overturn. Its central claim is counterintuitive: for generative tasks, deleting experts beats averaging them, and the reason is a subtle failure mode in merging that pruning avoids entirely.

Why merging quietly breaks: functional subspace collapse

Here is the problem with averaging two experts. In the original model, the router does not just decide whether to use expert i or expert j. On every token, it assigns them different, input-dependent weights. On one token it might lean 70/30 toward i, on the next 20/80. That shifting mixture is part of how the model computes. The router has independent, per-token control over each expert's contribution.

Merge i and j into a single expert and that control is gone. The router can now only turn the merged expert up or down as a whole. The dynamic, input-dependent blend between the two original behaviors has been flattened into one fixed compromise. The REAP authors call this functional subspace collapse: a rich, input-conditioned mixing policy is forced into a single static convex combination, and the model can never recover the distinctions it used to make.

The paper makes this precise. Merging two experts introduces an irreducible error whose size is governed by three factors:

$\text{Error} \;\propto\; \mathbb{E}\!\left[(g_i + g_j)^2\right] \cdot \mathrm{Var}\!\left[r(x)\right] \cdot \lVert \Delta_{ij} \rVert^2$

Reading it left to right: the error grows with the combined gate mass the router puts on the two experts $(g_i + g_j)$, with the variance of the input-dependent mixing ratio $r(x)$ between them, and with the functional gap $\Delta_{ij}$ between what the two experts actually compute. The key consequence is the middle term. As long as the router's policy varies across inputs (so its variance is above zero) and the experts are not literally identical, the merge incurs strictly positive excess error that no amount of clever averaging can remove. You are not losing a little precision at the margins. You are erasing a capability the router was actively using.

Pruning sidesteps this entirely. When you delete an expert, you do not touch the ones that remain. Their weights are untouched, and the router keeps its full independent control over each survivor. There is no forced compromise, no collapsed subspace. The only cost is whatever the deleted expert would have contributed, which is precisely the thing you want to minimize when you choose what to cut.

The REAP saliency criterion

So pruning is safe if, and only if, you delete the right experts. This is where REAP's actual mechanism lives, and it is refreshingly simple.

For each expert, REAP computes a saliency score that estimates how much the layer's output would suffer if that expert disappeared. A good score has to capture two independent things. First, how often and how strongly does the router select this expert? An expert the router almost never gates is cheap to lose. Second, when the expert does run, how large is its output? An expert that fires often but produces near-zero vectors is not doing much either. You only want to keep an expert that is both frequently chosen and impactful when chosen.

REAP multiplies exactly these two quantities. Over a calibration set of tokens, the saliency of expert $j$ is:

$S_j \;=\; \frac{1}{\lvert \mathcal{X}_j \rvert} \sum_{x \in \mathcal{X}_j} g_j(x)\,\bigl\lVert f_j(x) \bigr\rVert_2$

Here $g_j(x)$ is the router gate value assigned to expert $j$ for token $x$, $\lVert f_j(x) \rVert_2$ is the L2 norm (the magnitude) of that expert's output, and $\mathcal{X}_j$ is the set of calibration tokens where expert $j$ landed in the top-K routing selection. The score is the average, over the tokens that actually used the expert, of gate value times output size.

That product is the whole idea. Neither factor alone is enough. Gate value without output norm would keep experts the router likes but that contribute little. Output norm without gate value would keep experts that shout loudly on the rare occasions they run. Multiplying them ranks experts by their genuine average contribution to the layer. REAP prunes the experts with the lowest scores, layer by layer, and rescales the surviving router weights so the remaining experts still sum correctly. The name is literal: the score is the router-weighted expert activation, and pruning removes the bottom of that ranking.

One-shot, no retraining

The other thing worth stressing is what REAP does not require. It is a one-shot method. You run a modest calibration set through the model once to gather the gate values and activation norms, rank the experts, delete the low-saliency ones, and you are done. There is no gradient descent, no fine-tuning, no distillation to recover lost quality.

The calibration data is small and deliberately diverse. The released pipeline uses on the order of 24,000 samples at a maximum sequence length of 16,384 tokens, mixing general coding, reasoning over code, math and science, single-turn tool calling, and multi-turn agentic traces. The point of that mixture is to measure expert usefulness across the kinds of work the model is actually meant to do, so an expert that only matters for, say, tool calling is not accidentally scored as useless because the calibration set was all prose.

What the numbers say

The paper evaluates REAP across models from about 22 billion to over one trillion parameters. Two results carry most of the weight, and the first is easiest to see:

Pruning beats merging, and the gap widens as you compress harder: at 25 percent compression REAP's mean coding drop is about 2.8 percent versus more than 5 for merging; at 50 percent, merging loses over 20 percent while REAP's mean decrease stays at 1.9 (about 1.1 on math). The most dramatic case is creative writing on Qwen3-30B, where REAP retains 0.718 but one merging method collapses to 0.008. That collapse is functional subspace collapse showing up in a benchmark.

Second, the bigger the base model, the smaller the damage. Qwen3-Coder-480B pruned by half keeps non-agentic coding at 0.644, roughly a 1.4 percent decrease, and the paper calls code generation and tool calling for Qwen3-Coder-480B and Kimi-K2 near-lossless at that compression. Large MoE models simply carry more redundant capacity.

One honest caveat: multiple-choice knowledge benchmarks lose more, around 4 percent at 25 percent compression and 13 percent at 50. Deleting experts does remove knowledge; factual recall feels it more than generative coding does. REAP is a memory-for-breadth trade, not a free lunch. The paper was accepted to ICLR 2026 and the code is open on GitHub.

From paper to practice: the released checkpoints

The reason this method is worth your attention is that it has already left the lab. Cerebras maintains a growing collection of REAP-compressed checkpoints on Hugging Face, published as drop-in models rather than research artifacts. A representative slice:

The GLM-4.6 checkpoint pruned by 40 percent is a good stress test, because 40 percent is aggressive. It drops from 355B to 218B parameters, cutting each layer's routed experts from 160 to 96 while keeping 8 active per token. On the model card, it reports HumanEval at 95.1 against a 96.3 baseline, MBPP at 89.4 against 87.6, and MATH-500 (with thinking) at 93.3 against 95.5. In other words, after deleting two of every five experts, coding scores land within roughly a point of the original and one benchmark even ticks up inside the noise. The smaller Qwen3-Coder-REAP-25B-A3B, pruned 20 percent from the 30B coder, tells the same story: HumanEval moves from 92.1 to 94.5 and MBPP from 87.6 to 87.3.

Beyond GLM and Qwen, the collection includes MiniMax-M2-REAP-172B-A10B, Kimi-Linear-REAP-35B-A3B-Instruct, and a DeepSeek-V3.2 variant, plus community derivatives that add quantization on top, such as FP8 and GGUF builds. Because the pruned models keep the original architecture minus some experts, many run in vanilla vLLM with no custom code, which is much of the practical appeal.

A note on GLM versions

To be precise: Cerebras's official REAP checkpoints target the GLM-4 series (GLM-4.5-Air, GLM-4.6, GLM-4.7). The community has already taken the method to the 5-series though: 0xSero's GLM-5.2-REAP-504B prunes the 753B GLM-5.2 down to 504B with router knowledge distillation on top, shipped as GGUF for llama.cpp, and pipenetwork publishes 25, 37, and 50 percent prunes of the same base. The method is architecture-general, and the open code means new releases get pruned within days.

Who should care, and how to use it

REAP is most useful if you are trying to run a frontier open MoE model on hardware that cannot quite hold the full version. Concretely:

• You have one big GPU, not eight. A 40 percent prune of a 355B model can be the difference between fitting on a single 80 GB card and needing a second one. Reach for a REAP checkpoint before you reach for heavy quantization, or stack the two.
• Your workload is generative, not trivia. REAP shines on coding, tool calling, and agentic tasks, which is exactly where the released Qwen3-Coder and GLM checkpoints are aimed. If your product is a coding assistant, the accuracy loss is close to noise. If it is a factual question-answering system leaning on the model's world knowledge, test carefully, because knowledge benchmarks degrade faster.
• You want a drop-in, not a project. Because pruning is one-shot and changes nothing but expert count, these models behave like ordinary checkpoints. There is no new runtime to learn.

Using them is mostly picking a point on the curve: take the lightest checkpoint whose benchmarks clear your bar, prefer a big base pruned hard over a small base pruned lightly, and validate on your own data since your task mix may lean on knowledge more than the calibration set did. The open code also lets you prune any MoE with your own calibration data when your domain is narrow.

Takeaways

MoE models are memory-heavy by construction: they load every expert to use only a few. REAP reclaims that memory by permanently deleting the experts that contribute least, scored by a simple product of router gate value and output magnitude, averaged over a small calibration set. It works in one shot with no retraining.

The core insight is why deletion beats the more popular strategy of merging. Averaging experts destroys the router's per-token control over their blend, a failure the authors formalize as functional subspace collapse with a strictly positive, irreducible error. Pruning leaves the survivors and their routing untouched, so the only cost is the contribution of what you cut.

The evidence backs the theory. Across models from 22B to over a trillion parameters, REAP loses far less accuracy than merging, and the losses shrink as base models grow, reaching near-lossless coding and tool-calling on Qwen3-Coder-480B and Kimi-K2 at 50 percent compression. Most importantly, this is not a paper you have to reimplement: GLM-4.5-Air, GLM-4.6, GLM-4.7, Qwen3-Coder, and others already ship as pruned checkpoints you can download and serve today, trading a sliver of accuracy for a large cut in the memory it takes to run them.

References

1. Lasby, M., Lazarevich, I., Sinnadurai, N., Lie, S., Ioannou, Y., and Thangarasa, V. (2025). "REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression." arXiv:2510.13999. arXiv abstract and full HTML

2. Cerebras Research. "REAP: Router-weighted Expert Activation Pruning for SMoE compression" (code repository). GitHub

3. Cerebras. "Cerebras REAP" model collection. Hugging Face

4. Cerebras. "GLM-4.6-REAP-218B-A32B-FP8" model card (40% pruning of GLM-4.6). Hugging Face

5. Cerebras. "GLM-4.7-REAP-268B-A32B" model card (25% pruning of GLM-4.7). Hugging Face

6. Cerebras. "Qwen3-Coder-REAP-25B-A3B" model card (20% pruning of Qwen3-Coder-30B). Hugging Face

7. Cerebras. "Kimi-Linear-REAP-35B-A3B-Instruct" model card. Hugging Face

8. Cerebras. "MiniMax-M2-REAP-172B-A10B" model card. Hugging Face

9. 0xSero. "GLM-5.2-REAP-504B-GGUF" model card (community REAP prune of GLM-5.2 with router knowledge distillation). Hugging Face