Mixture of Experts
Swap one shared feed-forward layer for eight specialists plus a router that wakes only two per token. Total knowledge grows with all eight while each token pays for two.
Why Does This Exist?
Making a dense transformers model smarter by widening every layer raises compute for every token. Double the width and you roughly double the math per token, so scaling hits a budget wall long before quality stops improving.
Mixture of Experts breaks that link. Replace the single feed forward networks block with expert blocks plus a small router. Each token visits only experts, so total weights grow with while math per token grows with . With and , the layer holds eight times the weights at about twice the math. This page covers where the block sits, how the router blends outputs, and the two failure modes that appear at scale. The compression trick that later squeezes such models down lives in knowledge distillation.
One basic makes the rest click. A router is one linear map plus softmax over experts, producing scores through that sum to one. Top- keeps the largest scores and skips the rest entirely for that token.
Think of It Like This
A clinic with specialists and one triage desk
A small clinic has one general doctor who sees every patient. A larger clinic has eight specialists plus one triage nurse. Each patient stops at triage, gets sent to one or two relevant specialists, and never sees the other six.
Total medical knowledge grows with all eight specialists, while each patient pays for only one or two visits. Triage is the router, each specialist is an expert, and waking only two per patient is top-2 routing.
Where it stops: human specialists arrive pre-trained with known skills, while experts start identical and must divide the work through training. Triage can also overload one popular doctor, which is exactly the collapse failure covered below.
How It Actually Works
A transformer block has attention then a feed-forward block. Mixture of Experts keeps attention dense and swaps only the feed-forward block for an expert pool plus router. Every token passes attention normally, gets scored by the router, runs through its chosen experts, and returns a weighted blend:
Here is the token vector, is its router score for expert , and is that expert block. Unchosen experts run zero math for this token. Specialization is not assigned. One expert may end up favoring code tokens and another favoring dates simply because sending those tokens there lowered loss during training.
Worked example: two awake experts out of eight
Take and with width 64 to 256. Each expert holds weights. Totals are:
Active per token are:
Capacity quadruples at fixed per-token math, since . Now the blend. Router scores four experts as and keeps the top two. Their mass is . Renormalize:
Expert one outputs and expert two outputs . Blend:
That is the diagram headline: eight slots on show, two lit, and one blended vector leaving the layer.
Code
The point is the check: rebuild the parameter totals and the renormalized blend from the worked example.
d_model: int = 64d_ff: int = 256n_experts: int = 8k: int = 2
per_expert: int = d_model * d_ff + d_ff * d_modeltotal: int = n_experts * per_expertactive: int = k * per_expert
print(total) # -> 262144print(active) # -> 65536print(total // active) # -> 4
g1: float = 0.50g2: float = 0.30mass: float = g1 + g2w1: float = g1 / massw2: float = g2 / massprint(round(w1, 3), round(w2, 3)) # -> 0.625 0.375
out0: float = w1 * 4.0 + w2 * 0.0out1: float = w1 * 0.0 + w2 * 8.0print(round(out0, 1), round(out1, 1)) # -> 2.5 3.0The printout matches the hand computation: 262144 total weights, 65536 active per token, ratio 4, renormalized gates 0.625 and 0.375, blended output 2.5 and 3.0. Switch to 1 and active halves while the blend uses only the top gate.
Watch Out For
One expert grabbing every token
Without a balancing push, early luck sends more tokens to one expert, which then learns faster and grabs even more. The rest starve untrained while the router looks busy.
Symptom: routing histograms show one expert near 90 percent and several near zero. Fix: add the auxiliary load-balancing loss that rewards even splits, and watch per-expert counts during training, not just total loss.
Budgeting memory by active weights alone
All eight experts must sit in memory even though only two run per token. A model with 46.7B total weights but 12.9B active per token still needs room for 46.7B at serving time.
Symptom: math fits on one card while loading fails out of memory. Fix: plan memory by total weights, plan speed by active weights, and pair large pools with sharding from distributed training when one card cannot hold all experts.
The Quick Version
- Mixture of Experts swaps one feed-forward block for 8 experts plus a router that wakes 2 per token.
- Eight experts at 32768 weights each hold 262144 total while each token spends 65536, a 4 to 1 capacity ratio.
- Router scores 0.50 and 0.30 share mass 0.80, so they renormalize to 0.625 and 0.375.
- Expert outputs 4, 0 and 0, 8 blend through those gates into 2.5 and 3.0.
- Specialization emerges from training without any manual assignment of topics to experts.
- Unbalanced routing starves experts and total memory must hold all 8, not just the active 2.