Mixture of experts, explained: why 120B can run like 5B

Many of today’s largest open models carry two numbers in their names: Nemotron 3.5 Lightning 30B-A3B, Gemma 4 26B A4B, Nemotron 3 Ultra 550B-A55B. They are mixture-of-experts (MoE) models, and the two numbers are the point of the design: a model can hold a very large number of parameters while using only a small fraction of them for each token. Here is how that works, what it buys and what it costs.

Dense models use everything, every time

In a standard, dense transformer, every parameter takes part in processing every token. A dense 9B model does about 9 billion parameters’ worth of multiplication per token, every time. Making it know more means making it bigger, and making it bigger makes every token slower and more expensive, in proportion.

The idea: many experts, a few at a time

A mixture-of-experts model replaces some of the network’s layers, usually the feed-forward layers inside each transformer block, with a set of parallel copies called experts, plus a small router. For each token, the router scores the experts and sends the token through only the top few, typically one, two or four out of dozens or more than a hundred. Their outputs are combined, weighted by the router’s scores, and the rest of the experts sit idle for that token.

So the model has two sizes:

• Total parameters: everything stored, all experts included. This is the first number in 30B-A3B.
• Active parameters: what one token actually passes through, the shared layers plus the chosen experts. This is the “A” number.

The idea is older than today’s language models. Shazeer et al. (2017) showed that a sparsely gated layer with thousands of experts could grow a network’s capacity more than 1,000-fold with only minor losses in computational efficiency. Google’s Switch Transformer (2021) simplified routing to a single expert per token and scaled to over a trillion parameters.

Two well-documented examples

Mixtral 8x7B (Mistral AI, 2024) has eight experts in each layer and routes every token through two of them. It holds 47B parameters in total but uses 13B per token. Its name invites a misreading: 8 × 7 isn’t 56, because the experts share the attention layers and only the feed-forward parts are duplicated.

gpt-oss-120b (OpenAI, 2025) has 117B parameters in total and 5.1B active per token: each of its layers holds 128 experts, and each token goes through four. It is in LE PEAK, free on OpenRouter and on Ollama Cloud with an account.

Reading the names of the other MoE models in LE PEAK the same way:

What MoE buys

• Knowledge per unit of compute. More total parameters give more capacity to store facts and skills, while the cost per token tracks the active count. A 30B-A3B model aims for the knowledge of a mid-sized model at roughly the per-token compute of a 3B one.
• Speed. Generating a token mostly means reading weights from memory, and an MoE model reads only the active ones for each token. With one conversation at a time, a 120B-A12B model can generate tokens about as fast as a dense 12B model on the same hardware, given enough memory to hold it.
• Cheaper training for the same quality. Training cost also tracks active parameters, which is why many of the largest open models released since 2024 use MoE.

What MoE costs

• Memory for everything. Every expert must be loaded, because any token might need any expert. A 550B-A55B model needs the memory of a 550B model: about 1.1 TB at 16-bit precision, many GPUs’ worth, even though it computes like a 55B one. For running on your own hardware, total parameters are what decide whether a model fits; the memory arithmetic is in Why 9B?
• Harder serving. Experts are spread across GPUs, and tokens have to be shuffled between them as the router sends them around. Under heavy load, different tokens in a batch want different experts, so more of the model is read per step and the speed advantage shrinks.
• Balancing. Left alone, a router tends to favor a few experts and starve the rest. Training adds a load-balancing objective, and sometimes capacity limits per expert, to keep work spread out.
• Fine-tuning quirks. Adapting an MoE model is more delicate than a dense one: routing can shift, and some experts may barely see the new data.

Do experts specialize?

Not in the way the name suggests. There is no “law expert” or “biology expert” chosen by a person; routing is learned, and the experts divide work however training finds useful. When the Mixtral authors looked at which experts handled text from different domains, such as arXiv papers, biology and philosophy, they found no clear topical pattern. The clearer regularities were syntactic: some tokens, such as particular words in code or indentation, tended to go to the same experts.

Dense or MoE?

Neither design wins everywhere. MoE is attractive when you serve a large model to many users on big hardware and want the most quality per unit of compute. Dense models are simpler to serve, fine-tune and run on a single device, and their memory needs match their compute. KTwo Crust 1 Beta has 9 billion parameters; KTwo hasn’t published whether its architecture is dense or mixture-of-experts. The full spec is on the model page.

Frequently asked questions

What is a mixture-of-experts model?

A neural network in which some layers are made of many parallel “experts”, and a learned router sends each token through only a few of them. The model stores far more parameters than it uses for any one token.

What does 30B-A3B mean?

A mixture-of-experts model with about 30 billion parameters in total, of which about 3 billion are active for each token.

Is a mixture-of-experts model faster than a dense model?

Faster than a dense model of the same total size, because each token uses only the active parameters. But it needs the memory of its total size, and its advantage shrinks under heavy, batched load.

How much memory does an MoE model need?

Enough for all of its parameters, not just the active ones: about 2 bytes per parameter at 16-bit precision, 1 at 8-bit and 0.5 at 4-bit, plus memory for the conversation’s KV cache.

Is gpt-oss a mixture-of-experts model?

Yes. gpt-oss-120b has 117 billion parameters in total and 5.1 billion active per token, with 128 experts per layer and four used per token, according to OpenAI’s model card.