Mixture of Experts: Huge Models That Stay Fast

In the last lesson you saw the core tension in model design: big models are more capable but slow and expensive, while small models are fast and cheap but have less capacity. For years that looked like a straight tradeoff. Then a design called Mixture of Experts (MoE) offered a way to partly escape it: build a model with the enormous total capacity of a giant, but make it run at roughly the speed and cost of something much smaller. Most frontier models today use this idea.

You do not need any math to understand why it works. This lesson gives you the intuition, and the vocabulary (active versus total parameters, routing) that lets you read modern model announcements accurately.

What You'll Learn

• The problem MoE solves
• What "experts" and a "router" are, in plain terms
• The crucial difference between total and active parameters
• Why frontier labs adopted MoE and what it means for you

The idea: don't use the whole brain for every word

A traditional model is dense: every parameter is used to process every single token. If it has 70 billion parameters, all 70 billion do work for every token you send. That is thorough, but wasteful. Not every part of a giant model is relevant to every token.

Mixture of Experts takes a different approach. Inside the model, certain layers are split into many smaller sub-networks called experts. For any given token, a small traffic-controller called the router picks just a few experts to handle it and skips the rest. The experts are not neatly labeled "the poetry expert" or "the math expert" (the specialization that emerges is subtle and not human-readable), but the effect is that each token only activates a slice of the model, not the whole thing.

Because most experts sit idle for any given token, the model does far less computation per token than its size would suggest, while still having a huge pool of specialized capacity to draw from.

Total vs active parameters: the number that matters

This is the single most important idea in the lesson, and the one most misread in headlines.

• Total parameters = every parameter across all experts and shared parts. This is the model's full capacity, the big impressive number.
• Active parameters = only the parameters actually used for a given token (the shared parts plus the handful of experts the router picked). This is what determines speed and cost per token.

In an MoE model, active parameters are a small fraction of total parameters. Illustratively, a model might have many experts in a layer but activate only two of them per token. The result is a model that stores the knowledge of a giant but does the per-token work of something much smaller.

Notice one honest catch in that table: an MoE model still has to fit in memory in full, because the router might call on any expert at any moment. So MoE saves on computation and speed, but not on the memory needed to hold the whole model. That is why frontier MoE models still run in data centers, not on your phone, even though their per-token compute is modest.

Why frontier labs use MoE

Put the pieces together and the appeal is obvious. MoE lets a lab:

• Scale up total capacity (more experts, more knowledge) without a proportional increase in the compute spent per token.
• Serve responses faster and cheaper than a dense model of the same total size, because only active parameters do work.
• Add specialization through many experts, so different kinds of input can be handled by the best-suited slices of the model.

That combination (giant capacity, modest per-token cost) is exactly what you want when you are serving millions of users. It is why so many of today's most capable models are MoE under the hood, even when the marketing does not dwell on it.

What this means when you read model announcements

Two practical habits will make you read AI news more accurately:

• When you see two parameter numbers, find out which is total and which is active. A phrase like "big total parameters, far fewer active" is the tell-tale sign of an MoE model. The active number tells you more about speed and cost; the total number tells you more about capacity.
• Do not compare a dense model's parameter count directly against an MoE model's total count. They mean different things. A dense model uses all its parameters per token; an MoE model does not. Comparing the raw totals is apples to oranges.

You will not usually choose MoE versus dense yourself, that is the model maker's decision. But understanding it explains why modern models can be simultaneously enormous and reasonably fast, and it inoculates you against headlines that quote the biggest number without context.

One related term you will keep seeing in those announcements: open weights versus closed models. Open-weight models publish their parameters (the "weights" in the name; the two words mean the same thing) so anyone can download and run them on their own hardware; closed models are reachable only through the maker's app or API. MoE has made open weights far more practical, because a model with a huge total parameter count but a small active count can still run at a reasonable cost, which is why many of the strongest openly downloadable models today are MoE designs.

One distinction worth keeping straight, because press coverage muddles it constantly: open weights is not the same as open source. Most open releases publish the finished parameters and a usage license, not the training data or the recipe that produced them. You can run the model; you cannot rebuild it.

Key Takeaways

• A dense model uses every parameter for every token; Mixture of Experts uses only a chosen few sub-networks (experts) per token.
• A router decides which experts handle each token, leaving most experts idle for that token.
• Total parameters measure capacity (the big number); active parameters measure per-token speed and cost (the smaller number), and active is a fraction of total.
• MoE still must fit in memory in full, so it saves compute and speed, not the memory footprint.
• Frontier labs use MoE to get near-giant capability at much lower per-token cost, so read announcements by checking which parameter number is active versus total.