ML Evaluation Metrics Cheat Sheet
Which metric to trust for which problem — classification, regression, ranking, and generation, plus the traps in each.
- Precision / Recall
- F1 & ROC-AUC
- RMSE / MAE
- Perplexity
Classification
- Accuracy — fine only when classes are balanced.
- Precision — of predicted positives, how many were right. Optimize when false positives are costly.
- Recall — of actual positives, how many you caught. Optimize when false negatives are costly.
- F1 — harmonic mean of precision and recall; the single number for imbalanced data.
- ROC-AUC — threshold-independent ranking quality; can look rosy on skewed data (prefer PR-AUC there).
Regression
- MAE — average absolute error, robust to outliers.
- RMSE — penalizes large errors more; same units as the target.
- R² — fraction of variance explained; can be negative for bad models.
Generation / language
- Perplexity — how surprised the model is; lower is better, comparable only within the same tokenizer.
- BLEU / ROUGE — n-gram overlap; cheap but blind to meaning.
- Human / LLM-as-judge — the real signal for open-ended quality.
Traps
- A single held-out split hides variance — cross-validate.
- Leakage inflates every metric; check your feature pipeline.
- Always compare against a trivial baseline.