ZeRO
Zero Redundancy Optimizer is an advanced memory optimization technique that massively partitions model states across multiple GPUs to enable training giants.
Think of It Like This
Like tearing a massive textbook into chapters and giving one chapter to each student to hold, rather than forcing every student to carry the whole book.
In standard data parallelism, every GPU holds a full replica of the model weights, gradients, and optimizer states, wasting massive amounts of VRAM. ZeRO automatically partitions these states across the cluster. It allows researchers to train trillion-parameter models on standard hardware clusters by virtually pooling the VRAM of all available GPUs.