Data Parallelism: Copies That Share One Update
Each GPU holds a full copy of the model and trains on its own slice of data, then all copies average their gradients and take the same step.
Why Does This Exist?
One GPU can only chew through so many examples per step. You add three more GPUs to finish epochs sooner, but now you own a sync problem: four copies of the weights drift apart the moment they train on different rows.
You already know the single device loop from stochastic gradient descent: sample a small batch, estimate the gradient, nudge the weights. Data parallelism keeps that loop intact on each worker and adds one collective step. Average the four estimates before anyone moves.
This page covers that replica plus average plus shared step pattern. For the full map of splitting data, layers, tensors and optimizer state, start with distributed training. When one full copy no longer fits in memory, the sequel is zero-and-fsdp. When the model itself must be cut across devices, read tensor parallelism and pipeline parallelism.
Think of It Like This
The grading team
You and three friends must grade 400 quizzes by morning. You photocopy the answer key four times, hand each grader 100 different quizzes, and grade in parallel.
At the end of each round you huddle, average the four suggested rubric tweaks into one agreed tweak, and every copy of the key gets the same edit before the next round.
Where it stops: human graders tolerate a sloppy copy or a missed huddle. GPU replicas do not. If one copy starts from different weights or skips the average, the four models stop being one model after a single step.
How It Actually Works
Call the shared weights , with for the step. Call the step size , the replica count , and the per replica batch size . On replica , write for its local gradient estimate, meaning the average loss gradient over its examples. Write for the mean across replicas.
The update matches one big batch of size built from disjoint shards. That match is exact for plain summed gradients, which is why this pattern is also called synchronous SGD.
How does all-reduce share the mean?
Each replica holds its own . An all-reduce sum collects the vectors over the fast link, adds them, and returns the total to every replica. Each replica then divides by . No central server decides. Every copy forms the same and the same , so replicas stay identical when they started from the same and random ops share a seed.
Worked example: four numbers to one step
Take with scalar gradients to keep the arithmetic visible: , , , . Add them in order. , minus leaves , plus gives . Divide by :
Start from with :
Every copy applies . The diagram draws these same five numbers: four locals around the boxes, on the all-reduce bar, as the shared result. With per replica batch , this step behaves like one batch of .
Where does the batch size go?
Effective batch is . Double the replicas and you double the examples per update, which steadies the estimate but halves the steps per pass. That trade shapes batch size effects: past a point the steadier gradient stops helping test score at equal epochs.
Code
The point is the check: average the same four locals and apply the same step, then confirm and .
grads: list[float] = [0.8, 0.2, -0.4, 0.6]mean: float = sum(grads) / len(grads)w: float = 2.0lr: float = 0.1w_new: float = w - lr * meanprint(f"{mean:.2f} {w_new:.2f}") # -> 0.30 1.97The printout matches the hand math: mean , updated weight . Change one local to and you will see the mean move, which is the whole sync contract in one line.
Watch Out For
Large batch, flat test score
Symptom: you scale from 1 to 8 replicas, hold fixed, and training loss falls faster per step yet test score stalls or drops at equal epochs. The effective batch grew while the step size did not, so each update plays it too safe.
Fix: scale the step size with the replica count as a starting rule, add warmup for the first few epochs, and retune from there. Read batch size effects before you add more replicas.
Same rows on every replica
Symptom: four GPUs run, the fast link stays quiet, and the averaged gradient equals each local one. Your loader dealt the same shard to every worker, so you paid for four copies of one gradient.
Fix: shard the loader with a distributed sampler and set its epoch each pass so shards reshuffle without overlap. If you use batch normalization, sync the statistics across replicas, because per replica means on tiny shards bias the normalization. See batch normalization for why the stats must match.
The Quick Version
- Data parallelism keeps a full model copy on each GPU, feeds disjoint shards, averages local gradients with all-reduce, and applies one identical update.
- With locals , , and , the shared mean is , so stepped with becomes everywhere.
- Effective batch equals replicas times per replica batch, so behaves like a single batch of .
- It cannot train a model that exceeds one device memory, so oversized models need sharding in zero-and-fsdp or splitting in tensor parallelism.
- Scale the step size with warmup when you add replicas, and shard the loader so no two replicas repeat rows.