Skip to content
AI360XpertBeta
Core ML

Visual explainer

Mixture Of Experts

A router selects a few experts per token. Follow a four-expert example through gate weights, output blending, dispatch, load balance and capacity overflow.

Evaluating every expert sends the same token through four subnetworks.
Evaluating every expert sends the same token through four subnetworks.

A sparse mixture-of-experts layer routes each token representation to a subset of its expert subnetworks, then combines their outputs. This four-expert fixture first shows the comparison case: the same x is evaluated by all four. A conventional dense feed-forward layer is not literally a bank of four experts; this picture isolates the cost of evaluating all available experts before introducing selection.

Select experts for one token

The router selects the two largest scores for this token.
The router selects the two largest scores for this token.

Our synthetic router distribution is (0.50,0.30,0.12,0.08)(0.50,0.30,0.12,0.08) over E1–E4. The bars use 400 drawing units per probability unit. Top-two selection keeps E1 and E2, marked by solid outlines and the bracket. A learned router can select different experts for a different token; expert numbers do not imply fixed human-readable specialties. This is routing within one layer, not selecting independent chatbots.

Normalize the selected weights

Normalizing the selected mass changes the two weights to 0.625 and 0.375.
Normalizing the selected mass changes the two weights to 0.625 and 0.375.

For this example, explicitly normalize over the selected experts: retained mass is 0.5+0.3=0.80.5+0.3=0.8, giving weights 0.5/0.8=0.6250.5/0.8=0.625 and 0.3/0.8=0.3750.3/0.8=0.375. The 400-unit strips show the discarded 0.2 as a dashed tail, then the retained segments expanded to total one. Selected-logit softmax appears in the original sparse-gating formulation. Normalization is a design choice; other implementations retain the selected probabilities' original scale.

Check the batch load

Equal batch totals can hide very different loads across experts.
Equal batch totals can hide very different loads across experts.

Switch explicitly to top-one routing for an eight-token batch. The left assignment counts (5,1,1,1)(5,1,1,1) and the right (2,2,2,2)(2,2,2,2) both total eight. Every bar uses 30 drawing units per assignment. Load-balancing objectives can encourage more even traffic; the right panel is a comparison assignment, not a guaranteed training result. Equal token counts also do not guarantee identical device finish times or task quality.

Count work and storage separately

Selecting two experts halves expert evaluations while all four remain stored.
Selecting two experts halves expert evaluations while all four remain stored.

Return to the single-token top-two fixture. The four expert blocks remain in the model, while only E1 and E2 are active for this token. If their evaluation costs are equal, two evaluations use half the expert computation of four; the bars use 70 drawing units per evaluation. Router work, shared layers, dispatch, communication and hardware utilization are excluded. This does not promise half the total latency or a particular accuracy level.

Compare top one and two

Changing the selected expert count changes the output in this fixed example.
Changing the selected expert count changes the output in this fixed example.

Hold expert outputs fixed at E1(x)=(4,0)E_1(x)=(4,0) and E2(x)=(0,8)E_2(x)=(0,8). Under our selected-normalization convention, top one weights E1 by one and returns (4,0)(4,0). Top two returns 0.625(4,0)+0.375(0,8)=(2.5,3)0.625(4,0)+0.375(0,8)=(2.5,3). Both coordinate planes use 50 drawing units per component. This is an output difference, not a quality comparison. Switch routing instead retains a router gate factor for its selected expert, so its output need not equal the normalized top-one result shown here.

Watch for collapse

A collapsed top-one router can send the entire batch to one expert.
A collapsed top-one router can send the entire batch to one expert.

In a separate top-one batch, all eight assignments go to E1. At 30 drawing units per assignment, its bar is 240 units long and the others are zero. The unused experts receive no expert-task gradient from these tokens, although router auxiliary objectives can still have gradients. Routing collapse wastes the bank's available capacity; adding experts alone does not ensure that training uses them.

Compute the blended vector

Weighted expert vectors add to the token output 2.5 comma 3.
Weighted expert vectors add to the token output 2.5 comma 3.

Return to the top-two token. The white horizontal contribution is 0.625(4,0)=(2.5,0)0.625(4,0)=(2.5,0). The mint vertical contribution is 0.375(0,8)=(0,3)0.375(0,8)=(0,3), placed at the first contribution's tip. Their sum reaches (2.5,3)(2.5,3) along the dashed resultant. Both axes use 50 units per component. Experts must produce compatible output shapes; this layer combines vectors rather than voting over generated sentences.

Preserve token identity

Dispatch groups tokens by expert while gather restores their original positions.
Dispatch groups tokens by expert while gather restores their original positions.

Use two token identities A and B in top-one mode. Matrix rows preserve the original token order; columns identify E1 and E2. The marked cells send A to E2 and B to E1. Grouping by expert produces execution order B, A; gathering restores outputs to rows A, B. The right arrows denote that restoration, not all-to-all physical wires. Distributed expert execution may require moving token activations and returning results, which adds communication cost.

Where It Breaks

An expert can overflow even when the batch has enough total slots.
An expert can overflow even when the batch has enough total slots.

Return to the eight-token top-one load (5,1,1,1)(5,1,1,1). With two slots per expert, total capacity is eight but E1 receives five assignments. Only two fit at E1; three overflow while E2–E4 each leave a slot empty. Solid squares are served tokens, dashed squares unused slots, and amber circles overflow tokens. The bypass shows the Switch-style policy that skips expert computation for overflow and carries the representation through the residual route. It does not discard the token from the sequence. Other MoE designs can use different overflow policies. More capacity reduces this pressure but costs buffer space and potentially padded computation; the choice needs workload measurements.

The Quick Version

  • Evaluating every expert repeats subnetwork work for each token.
  • A router selects experts using token-dependent scores.
  • This example normalizes the selected weights to sum to one.
  • Equal batch totals can hide uneven expert loads.
  • Sparse activation saves expert work while retaining stored weights.
  • Changing expert count can change the output vector.
  • Collapsed routing can leave most experts unused.
  • Weighted expert vectors sum into one token output.
  • Gather restores the token positions changed by dispatch.
  • Per-expert limits can overflow despite enough total capacity.