Core ML

Fault-Tolerant Training

When you train a model on 10,000 GPUs for a month, it is mathematically guaranteed that hardware will break. Fault-Tolerant Training is how you ensure that a single broken cable doesn't waste three weeks of million-dollar compute.

In a massive cluster, hardware failures are inevitable. A robust fault-tolerant system periodically saves Checkpoints to stable storage, detects node crashes instantly, ejects the dead node, and rescales the remaining cluster elastically without human intervention.

Why Does This Exist?

If you train a small model on a single GPU for 3 hours and the machine crashes, you just reboot it and run the script again.

However, training a frontier Large Language Model involves orchestrating anywhere from 1,000 to 25,000 GPUs running in parallel for 3 to 6 months. At that scale, hardware failure is not a possibility—it is a statistical certainty. Every single day, something will break: a GPU will overheat, a motherboard will die, a PCIe riser will fail, or a network switch will drop packets.

When you are paying $\$ 100,000$ a day for cluster time, you cannot afford to have 9,999 GPUs sitting idle waiting for a human engineer to wake up, find the one dead GPU, restart the job, and manually rewrite the cluster configuration file. Fault-Tolerant Training is the infrastructure that automates this survival.

Think of It Like This

Imagine you are directing an orchestra of 10,000 musicians playing a continuous symphony for three months. Inevitably, someone's violin string will snap, or someone will pass out. If the entire orchestra had to stop, wait for a replacement musician, and then start playing the symphony from the very beginning, the concert would never finish.

Instead, you record the performance every 30 minutes (Checkpointing). When a violinist collapses, you instantly pause, pull them off stage, re-seat the remaining musicians (Elastic Rescaling), and have everyone resume playing from the last 30-minute recording.

The Three Pillars of Fault Tolerance

1. High-Frequency, Asynchronous Checkpointing

The core defense against failure is saving your progress. You must periodically save the model weights and the optimizer state to distributed storage (like an AWS S3 bucket).

However, saving 500GB of state across thousands of nodes takes time. If you freeze training for 5 minutes every hour just to write files to disk, your GPU utilization plummets. Modern frameworks use Asynchronous Checkpointing. The training loop copies the weights to a fast, local buffer in CPU RAM in milliseconds, and the GPUs immediately resume training. A background thread on the CPU slowly uploads that buffer to S3 over the network without blocking the GPUs.

2. Node Failure Detection and Ejection

When a GPU physically dies, the training job will hang. It usually hangs at an All-Reduce operation, because the remaining 9,999 GPUs are endlessly waiting for the dead GPU to send its gradients.

You need a health-monitoring layer (like Kubernetes, Slurm, or PyTorch Elastic/Torchrun). This layer constantly checks the heartbeat of all nodes. If a node stops responding, the orchestrator instantly kills the training job across all remaining nodes to prevent them from hanging indefinitely.

3. Elastic Rescaling

In the old days, if you started a job with 1,024 GPUs, and one died, you couldn't restart the job until you found a replacement GPU to bring the count exactly back to 1,024. Elastic Training solves this. Frameworks like torchrun dynamically adjust the world size. If 8 GPUs die, the orchestrator automatically rewrites the communication topology to expect exactly 1,016 GPUs. It loads the last checkpoint, distributes the data shards across the 1,016 remaining GPUs, and resumes training within minutes, completely automatically.

Show Me the Code

You rarely write the failure recovery logic into your model script. Instead, you launch your script using a fault-tolerant orchestrator like torchrun.

1# Instead of standard python train.py2# You use torchrun, which handles health checks and elastic restarts.3
4torchrun \5    --nnodes=128 \6    --nproc_per_node=8 \7    --max_restarts=10 \8    --rdzv_id=my_llama_training_job \9    --rdzv_backend=c10d \10    --rdzv_endpoint=master-node:29500 \11    train.py

If a node fails, torchrun tears down the process group, re-establishes the rdzv (rendezvous) with the surviving nodes, and restarts train.py. Your script's only job is to check for the latest checkpoint upon startup.

Watch Out For

The Silent Straggler. A GPU dying entirely is easy to handle; it triggers a clear error, the node is ejected, and training resumes. The true nightmare is the Straggler—a GPU that hasn't died, but has suffered a hardware degradation causing it to run at 20% speed. Because distributed training requires GPUs to synchronize at every layer, the entire cluster of 10,000 GPUs is forced to run at the speed of the single slowest GPU. You must build strict timeout monitoring to detect and intentionally kill stragglers so the cluster can replace them.

The Quick Version

At massive scales (thousands of GPUs), hardware failures happen daily.
Asynchronous Checkpointing saves the model state to stable storage in the background without forcing the expensive GPUs to wait.
When a node fails, the orchestrator detects the dropped heartbeat and kills the hanging job.
Elastic Rescaling allows the cluster to automatically restart the training job using the surviving nodes, adjusting the tensor and data sharding dynamically.
Stragglers (slow GPUs) are worse than dead GPUs because they bottleneck the entire cluster without throwing an error.

Related concepts

Training Cost Estimation