Visual explainer
Mixture Of Experts
A router selects a few experts per token. Follow a four-expert example through gate weights, output blending, dispatch, load balance and capacity overflow.
A sparse mixture-of-experts layer routes each token representation to a subset of its expert subnetworks, then combines their outputs. This four-expert fixture first shows the comparison case: the same x is evaluated by all four. A conventional dense feed-forward layer is not literally a bank of four experts; this picture isolates the cost of evaluating all available experts before introducing selection.
Select experts for one token
Our synthetic router distribution is over E1–E4. The bars use 400 drawing units per probability unit. Top-two selection keeps E1 and E2, marked by solid outlines and the bracket. A learned router can select different experts for a different token; expert numbers do not imply fixed human-readable specialties. This is routing within one layer, not selecting independent chatbots.
Normalize the selected weights
For this example, explicitly normalize over the selected experts: retained mass is , giving weights and . The 400-unit strips show the discarded 0.2 as a dashed tail, then the retained segments expanded to total one. Selected-logit softmax appears in the original sparse-gating formulation. Normalization is a design choice; other implementations retain the selected probabilities' original scale.
Check the batch load
Switch explicitly to top-one routing for an eight-token batch. The left assignment counts and the right both total eight. Every bar uses 30 drawing units per assignment. Load-balancing objectives can encourage more even traffic; the right panel is a comparison assignment, not a guaranteed training result. Equal token counts also do not guarantee identical device finish times or task quality.
Count work and storage separately
Return to the single-token top-two fixture. The four expert blocks remain in the model, while only E1 and E2 are active for this token. If their evaluation costs are equal, two evaluations use half the expert computation of four; the bars use 70 drawing units per evaluation. Router work, shared layers, dispatch, communication and hardware utilization are excluded. This does not promise half the total latency or a particular accuracy level.
Compare top one and two
Hold expert outputs fixed at and . Under our selected-normalization convention, top one weights E1 by one and returns . Top two returns . Both coordinate planes use 50 drawing units per component. This is an output difference, not a quality comparison. Switch routing instead retains a router gate factor for its selected expert, so its output need not equal the normalized top-one result shown here.
Watch for collapse
In a separate top-one batch, all eight assignments go to E1. At 30 drawing units per assignment, its bar is 240 units long and the others are zero. The unused experts receive no expert-task gradient from these tokens, although router auxiliary objectives can still have gradients. Routing collapse wastes the bank's available capacity; adding experts alone does not ensure that training uses them.
Compute the blended vector
Return to the top-two token. The white horizontal contribution is . The mint vertical contribution is , placed at the first contribution's tip. Their sum reaches along the dashed resultant. Both axes use 50 units per component. Experts must produce compatible output shapes; this layer combines vectors rather than voting over generated sentences.
Preserve token identity
Use two token identities A and B in top-one mode. Matrix rows preserve the original token order; columns identify E1 and E2. The marked cells send A to E2 and B to E1. Grouping by expert produces execution order B, A; gathering restores outputs to rows A, B. The right arrows denote that restoration, not all-to-all physical wires. Distributed expert execution may require moving token activations and returning results, which adds communication cost.
Where It Breaks
Return to the eight-token top-one load . With two slots per expert, total capacity is eight but E1 receives five assignments. Only two fit at E1; three overflow while E2–E4 each leave a slot empty. Solid squares are served tokens, dashed squares unused slots, and amber circles overflow tokens. The bypass shows the Switch-style policy that skips expert computation for overflow and carries the representation through the residual route. It does not discard the token from the sequence. Other MoE designs can use different overflow policies. More capacity reduces this pressure but costs buffer space and potentially padded computation; the choice needs workload measurements.
The Quick Version
- Evaluating every expert repeats subnetwork work for each token.
- A router selects experts using token-dependent scores.
- This example normalizes the selected weights to sum to one.
- Equal batch totals can hide uneven expert loads.
- Sparse activation saves expert work while retaining stored weights.
- Changing expert count can change the output vector.
- Collapsed routing can leave most experts unused.
- Weighted expert vectors sum into one token output.
- Gather restores the token positions changed by dispatch.
- Per-expert limits can overflow despite enough total capacity.