Expert Parallelism
A scaling technique for mixture of experts models where neural network experts are distributed across separate GPUs to increase computational efficiency.
Think of It Like This
Like managing a massive hospital by sending heart patients to the cardiology floor and brain patients to neurology so no single doctor gets overwhelmed.
In a Mixture of Experts (MoE) architecture, activating all parameters for every token is highly inefficient. Expert parallelism solves this by distributing expert subnetworks across multiple devices. The routing mechanism sends tokens over the network to the appropriate device, allowing for massive scaling of model capacity without a proportional increase in compute.