Skip to content
AI360XpertBeta

Mixture of Experts

Swap one shared feed-forward layer for eight specialists plus a router that wakes only two per token. Total knowledge grows with all eight while each token pays for two.

Eight experts hold 262144 weights while the router wakes two per token for 65536 active, blending 0.50 and 0.30 into 2.5 and 3.0
Eight experts hold 262144 weights while the router wakes two per token for 65536 active, blending 0.50 and 0.30 into 2.5 and 3.0

Why Does This Exist?

Making a dense transformers model smarter by widening every layer raises compute for every token. Double the width and you roughly double the math per token, so scaling hits a budget wall long before quality stops improving.

Mixture of Experts breaks that link. Replace the single feed forward networks block with NN expert blocks plus a small router. Each token visits only kk experts, so total weights grow with NN while math per token grows with kk. With N=8N = 8 and k=2k = 2, the layer holds eight times the weights at about twice the math. This page covers where the block sits, how the router blends outputs, and the two failure modes that appear at scale. The compression trick that later squeezes such models down lives in knowledge distillation.

One basic makes the rest click. A router is one linear map plus softmax over NN experts, producing scores g1g_1 through gNg_N that sum to one. Top-kk keeps the kk largest scores and skips the rest entirely for that token.

Think of It Like This

A clinic with specialists and one triage desk

A small clinic has one general doctor who sees every patient. A larger clinic has eight specialists plus one triage nurse. Each patient stops at triage, gets sent to one or two relevant specialists, and never sees the other six.

Total medical knowledge grows with all eight specialists, while each patient pays for only one or two visits. Triage is the router, each specialist is an expert, and waking only two per patient is top-2 routing.

Where it stops: human specialists arrive pre-trained with known skills, while experts start identical and must divide the work through training. Triage can also overload one popular doctor, which is exactly the collapse failure covered below.

How It Actually Works

A transformer block has attention then a feed-forward block. Mixture of Experts keeps attention dense and swaps only the feed-forward block for an expert pool plus router. Every token passes attention normally, gets scored by the router, runs through its kk chosen experts, and returns a weighted blend:

y=∑i∈top−kgi⋅FFNi(x)y = \sum_{i \in \mathrm{top-}k} g_i \cdot \mathrm{FFN}_i(x)

Here xx is the token vector, gig_i is its router score for expert ii, and FFNi\mathrm{FFN}_i is that expert block. Unchosen experts run zero math for this token. Specialization is not assigned. One expert may end up favoring code tokens and another favoring dates simply because sending those tokens there lowered loss during training.

Worked example: two awake experts out of eight

Take N=8N = 8 and k=2k = 2 with width 64 to 256. Each expert holds 64×256+256×64=32,76864 \times 256 + 256 \times 64 = 32{,}768 weights. Totals are:

8×32,768=262,1448 \times 32{,}768 = 262{,}144

Active per token are:

2×32,768=65,5362 \times 32{,}768 = 65{,}536

Capacity quadruples at fixed per-token math, since 262,144/65,536=4262{,}144 / 65{,}536 = 4. Now the blend. Router scores four experts as [0.50,0.30,0.12,0.08][0.50, 0.30, 0.12, 0.08] and keeps the top two. Their mass is 0.50+0.30=0.800.50 + 0.30 = 0.80. Renormalize:

0.50/0.80=0.625,0.30/0.80=0.3750.50 / 0.80 = 0.625, \quad 0.30 / 0.80 = 0.375

Expert one outputs [4,0][4, 0] and expert two outputs [0,8][0, 8]. Blend:

0.625×[4,0]+0.375×[0,8]=[2.5,3.0]0.625 \times [4, 0] + 0.375 \times [0, 8] = [2.5, 3.0]

That [2.5,3.0][2.5, 3.0] is the diagram headline: eight slots on show, two lit, and one blended vector leaving the layer.

Code

The point is the check: rebuild the parameter totals and the renormalized blend from the worked example.

d_model: int = 64d_ff: int = 256n_experts: int = 8k: int = 2
per_expert: int = d_model * d_ff + d_ff * d_modeltotal: int = n_experts * per_expertactive: int = k * per_expert
print(total)  # -> 262144print(active)  # -> 65536print(total // active)  # -> 4
g1: float = 0.50g2: float = 0.30mass: float = g1 + g2w1: float = g1 / massw2: float = g2 / massprint(round(w1, 3), round(w2, 3))  # -> 0.625 0.375
out0: float = w1 * 4.0 + w2 * 0.0out1: float = w1 * 0.0 + w2 * 8.0print(round(out0, 1), round(out1, 1))  # -> 2.5 3.0

The printout matches the hand computation: 262144 total weights, 65536 active per token, ratio 4, renormalized gates 0.625 and 0.375, blended output 2.5 and 3.0. Switch kk to 1 and active halves while the blend uses only the top gate.

Watch Out For

One expert grabbing every token

Without a balancing push, early luck sends more tokens to one expert, which then learns faster and grabs even more. The rest starve untrained while the router looks busy.

Symptom: routing histograms show one expert near 90 percent and several near zero. Fix: add the auxiliary load-balancing loss that rewards even splits, and watch per-expert counts during training, not just total loss.

Budgeting memory by active weights alone

All eight experts must sit in memory even though only two run per token. A model with 46.7B total weights but 12.9B active per token still needs room for 46.7B at serving time.

Symptom: math fits on one card while loading fails out of memory. Fix: plan memory by total weights, plan speed by active weights, and pair large pools with sharding from distributed training when one card cannot hold all experts.

The Quick Version

  • Mixture of Experts swaps one feed-forward block for 8 experts plus a router that wakes 2 per token.
  • Eight experts at 32768 weights each hold 262144 total while each token spends 65536, a 4 to 1 capacity ratio.
  • Router scores 0.50 and 0.30 share mass 0.80, so they renormalize to 0.625 and 0.375.
  • Expert outputs 4, 0 and 0, 8 blend through those gates into 2.5 and 3.0.
  • Specialization emerges from training without any manual assignment of topics to experts.
  • Unbalanced routing starves experts and total memory must hold all 8, not just the active 2.