Mixture of Experts
A neural network architecture where a gating network dynamically routes tokens to a sparse selection of specialized sub-networks during a forward pass.
Think of It Like This
Like a hospital triage system that routes a patient with a broken bone directly to orthopedics rather than a general doctor.
MoE drastically increases a model's total parameter count and capacity without proportionately increasing the compute required per token. For example, a model might have 8 experts but only activate 2 per token. It is highly effective but introduces significant complexity in load balancing and VRAM management during inference.