Debugging Training Runs
An ordered protocol for isolating why a training run is failing, moving from the simplest possible check to the more expensive ones rather than guessing.
Rather than a list of possible causes, it's a sequence: first confirm the model can drive the loss to near zero on a single batch, then check the loss at initialization against what random chance would give, then check tensor shapes and masks, then the learning rate, then numerical precision. Each step's result tells you whether to move to the next one or stop.
The ordering matters because later checks are more expensive to run and more ambiguous to read, so spending an hour tuning the learning rate before confirming the model can even overfit one batch is a common way to debug the wrong layer of the stack.