Large-Scale Training Infrastructure

Training infrastructure turns dozens of GPUs into one reliable machine with fast data loading, checkpoints, and automatic recovery from faults.

Sixty four workers draw from sharded data loaders into synchronous steps while half hourly checkpoints bound any fault to thirty lost minutes

Why Does This Exist?

One GPU trains a toy. A production backbone needs sixty four, and sixty four machines fail like sixty four machines: a dead NIC, a slow disk, one straggler holding every step hostage. Without infrastructure, a week-long run dies on day six and restarts from zero.

Training infrastructure makes the cluster act like one computer. Sharded loaders feed every worker, synchronous steps stay aligned, checkpoints bound losses, and health checks eject bad nodes fast. Efficient pretraining plans the budget, batching sizes the work, and data loaders own the feeding path.

Think of It Like This

A relay kitchen with ticket spikes

Sixty four cooks each own one dish pass, runners supply prepped ingredients, and completed tickets spike every minute. If one runner stalls, every cook waits. The head chef keeps a photo of each finished course so a dropped tray remakes one course, not the feast.

GPUs are cooks, loaders are runners, and checkpoints are the photos. The analogy stops at determinism: kitchens tolerate substitution, while training must resume bit-identical state to reproduce results.

How It Actually Works

Data shards split the dataset across workers with identical shuffling seeds per epoch. Each step runs forward and backward locally, then all-reduces gradients before the optimizer update, so every copy stays identical. Checkpoints write weights plus optimizer and loader positions on a schedule, and elastic runners replace dead workers from the latest checkpoint. Straggler detection compares step times and drops or rebalances slow nodes before they set the pace.

Worked example

Sixty four GPUs, global batch $2{,}048$ , per-GPU batch $32$ . One step takes $1.2$ seconds, so throughput is about $1{,}707$ images per second. Checkpoints every $30$ minutes cost $2$ minutes to write. A fault at hour $10$ loses at most $30$ minutes of work plus $3$ minutes to reload, against $10$ hours without checkpoints. The $2$ -minute tax buys back days.

Code

Throughput and worst-case loss from three cluster numbers.

1def cluster_math(gpus: int = 64, per_gpu: int = 32, step_s: float = 1.2) -> tuple[float, float]:2    """Images per second and minutes lost per fault at 30 min checkpoints."""3    global_batch = gpus * per_gpu  # images per synchronous step4    return global_batch / step_s, 30.0  # throughput, worst loss minutes5
6
7print(cluster_math())  # -> (1706.66..., 30.0)

Watch Out For

Letting one slow loader set the pace

Synchronous steps wait for the slowest worker, and slow storage on one node drags all sixty four. The symptom is high GPU count with single-GPU speed.

Pre-fetch aggressively, pin fast local disks, and alert on per-worker step times. Fix the tail before adding cards.

The Quick Version

Sharded loaders, synchronous all-reduce, checkpoints, and health checks form the stack.
Worked numbers: $64$ GPUs, batch $2{,}048$ , about $1{,}707$ images per second.
Half-hourly checkpoints bound any fault to $30$ lost minutes.
Stragglers set the pace in synchronous training, so watch per-worker times.
Reproduce from weights plus optimizer plus loader position, never weights alone.