Error Analysis
Read a sample of your model failures by hand, sort them into causes, and count each pile. The counts turn one opaque error rate into a ranked repair list.
Why Does This Exist?
Evaluation finishes and reports 92% accuracy. In a competition you celebrate. In production that number is where the work starts, because 8% failure on 2,500 backbone rows is 200 wrong answers shipping to users:
Without knowing why those 200 fail, every fix is a guess. Teams try a bigger architecture, a new optimizer, a tuned learning rate. That loop has a name, hopeful hyperparameter tuning, and it is an expensive way to stand still. If 90 of the 200 are mislabeled ground truth, no optimizer on earth helps. Only relabeling does.
Error analysis replaces the guessing with a ranked list. Read the failures, group them by cause, count the groups, fix the biggest one first.
Two ideas carry the rest. Accuracy is the share of right answers, and the confusion matrix shows where the wrong ones land. Classification metrics covers both. This page covers the manual loop that starts where the metric stops. The repair for shaky numbers on familiar inputs lives in calibration.
Think of It Like This
A doctor who counts fevers by cause
A thermometer says the patient has a fever. That number is real and useless on its own, because fever is a symptom, not a disease. Prescribing at random might work and might harm.
So the doctor diagnoses first. After examining a run of cases the pattern is 45% bacterial, 30% viral, 15% allergic, 10% something else. Treatment follows the counts: antibiotics for the big group, different plans for the rest. The counting is the medicine choosing itself.
The mapping is direct. The accuracy number is the thermometer, each failure is a patient, the cause groups are the diagnoses, and fixing the largest group is treating the most patients first.
Where it stops: a treated patient heals, while a fixed data bucket changes what the model learns next, so counts shift after every fix and the loop reruns. Doctors also get new tests on demand. You get the sample you sampled, so sampling it well is part of the skill.
How It Actually Works
Four steps, in order, with human judgment doing the work no script can.
Worked example: 200 failures on 2,500 rows
Sample the errors first. Run the model over the evaluation set, keep the 200 it got wrong, and read them. On a small set read all of them. On a huge set take a random 50 to 100, enough to show the dominant modes without taking weeks.
Then taxonomize: open each failure and ask why it went wrong, inventing buckets as patterns emerge. Night photos keep failing, so low light becomes a bucket. The model says positive, truth says negative, but reading the text shows the model was right and the annotator slipped, so mislabeled truth becomes a bucket. Suppose our 200 land like this:
Check the sum: . Now the shares:
That is the diagram: four ranked piles with the mislabeled pile marked as the fix. Act on the largest pile only. Those 90 mislabeled rows were predictions the model actually got right, so relabeling moves them from wrong to right. Correct rows go from 2,300 to , and accuracy goes from:
A 3.6 point gain with zero modeling. Then re-evaluate, resample, and loop. The smaller piles wait their turn.
Why volume order is the whole discipline
Every taxonomy hides one fascinating rare case that begs for a clever architecture. The 20 genuine failures here are 10% of the sample. Three weeks of modeling buys at most a tenth of the headroom, while the boring relabeling buys nearly half. Strict volume order is what keeps the exotic 10% from eating the schedule.
Code
The point is the tally: rebuild the worked example counts and confirm the shares plus the 95.6% payoff.
from collections import Counter
sample: list[str] = ["mislabeled"] * 90 + ["ambiguous"] * 60 + ["format"] * 30 + ["genuine"] * 20counts: Counter[str] = Counter(sample)total: int = len(sample)for bucket in ["mislabeled", "ambiguous", "format", "genuine"]: share: float = counts[bucket] / total print(bucket, counts[bucket], f"{share:.0%}") # -> mislabeled 90 45% # -> ambiguous 60 30% # -> format 30 15% # -> genuine 20 10%
print((2300 + 90) / 2500) # -> 0.956The four lines match the hand arithmetic: 90 at 45%, 60 at 30%, 30 at 15%, 20 at 10%. The final line confirms relabeling alone lifts 92.0% to 95.6%. Change one ambiguous row to genuine and watch only the third decimal move: small piles cannot move the headline.
Watch Out For
Automating the taxonomy before reading anything by hand
Clustering the failures or asking a language model to categorize them feels efficient and skips the actual skill. Automated buckets hallucinate causes, merge distinct problems with similar wording, and miss domain details a practitioner spots in seconds.
Symptom: neat buckets whose fixes do nothing, or a taxonomy nobody trusts enough to act on. Fix: read the first 50 failures yourself before any tooling touches them. Automation can scale a taxonomy you already understand. It cannot build one.
Chasing the exotic small pile
The 20 genuine failures contain the most interesting modeling problem and the least headroom. Ten percent is ten percent no matter how elegant the fix.
Symptom: weeks spent on a novel architecture for a rare edge while the 45% mislabeled pile sits untouched. Fix: work strictly down the counts. Fix the boring high volume piles first, and revisit the exotic ones only when they top the ranking.
The Quick Version
- One aggregate metric starts the work. Accuracy 92% on 2,500 rows means 200 failures, and nothing about causes.
- Read the failures, group them by cause, count the groups. Our 200 split 90, 60, 30, 20, which is 45, 30, 15, and 10 percent.
- Fix the largest pile first. Relabeling the 90 mislabeled rows lifts accuracy from 92.0% to 95.6% with zero modeling.
- Re-evaluate and loop after each fix, because counts shift as piles drain.
- Never automate the first pass and never chase a small exotic pile past a big boring one.