Cheat sheetCore Machine LearningProblem framing, generalization, classic supervised and unsupervised models, features, validation, and leakage. Core Machine Learning Problem framing, generalization, classic supervised and unsupervised models, features, validation, and leakage. 01Problem FramingSupervisedLearn a mapping from features X to labels y (classification or regression).UnsupervisedFind structure without labels (clustering, density, dimensionality reduction).Self-supervisedCreate labels from the data itself, such as masked or next-token prediction.Reinforcement learningLearn a policy that maximizes reward through interaction.Baseline firstA constant, rule, or linear model sets the bar every model must beat.1Define target2Collect data3Split4Baseline5Train + tune6Evaluate7Deploy + monitorThe ML workflow; most failures trace back to the first two steps02Bias, Variance, GeneralizationE[(y − ŷ)²] = Bias² + Variance + σ²σ² is irreducible noise; model choice trades the first twoSymptomDiagnosisRemediesHigh train and val errorUnderfitting (bias)Richer model, better featuresLow train, high val errorOverfitting (variance)More data, regularize, simplifyVal good, prod badDistribution shift or leakageRe-check splits and featuresRead the learning curveIf train and validation error converge high, add capacity. If they stay far apart, add data or regularization. 03Linear Regressionŷ = w^T x + bw = (X^T X)⁻¹ X^T yOrdinary least squares: the normal equation minimizes squared errorAssumptionsLinear relation, independent errors, constant variance, little multicollinearity.CoefficientsChange in ŷ per unit of a feature, holding the others fixed.ScalingNot needed for OLS itself; required for regularized or gradient-based fits.OutliersSquared loss is sensitive; consider Huber loss or robust regression.04RegularizationRidge: min ‖y − Xw‖² + λ‖w‖₂²Lasso: min ‖y − Xw‖² + λ‖w‖₁λ controls strength; standardize features firstRidge (L2)Shrinks all weights smoothly; stable with correlated features.Lasso (L1)Drives some weights to exactly zero, giving feature selection.Elastic NetMixes L1 and L2; keeps groups of correlated features together.05Logistic RegressionP(y = 1 | x) = σ(w^T x + b), σ(z) = 1 / (1 + e^(−z))Loss = −[y log p + (1 − y) log(1 − p)]OutputA calibrated-ish probability; pick the decision threshold from business costs.Odds ratioe^(wᵢ) is the multiplicative change in odds per unit of feature i.MulticlassSoftmax regression or one-vs-rest.06Distance and Kernel Modelsk-NNPredicts from the k closest training points; no training step, slow inference.Scaling mattersDistance-based models need standardized features.SVMMaximizes the margin between classes; only support vectors define the boundary.Kernel trickRBF or polynomial kernels fit nonlinear boundaries without explicit features.C parameterLarge C fits training points tightly; small C widens the margin.07Trees and EnsemblesDecision treeGreedy splits on Gini or entropy; interpretable, high variance alone.Random forestBagged trees on bootstrap samples with random feature subsets.Gradient boostingTrees fit sequentially to the residuals (gradients) of the previous ensemble.XGBoost / LightGBM / CatBoostFast, regularized boosting; strong default for tabular data.StackingA meta-model learns how to combine base model predictions.Gini = 1 − Σ pₖ²Fₘ(x) = Fₘ₋₁(x) + η · hₘ(x)η = learning rate; hₘ = tree fit to the current gradients08Naive BayesP(y | x) = P(x | y) · P(y) / P(x)Naive assumptionFeatures are conditionally independent given the class.StrengthsVery fast, works with little data, strong text baseline.SmoothingLaplace smoothing avoids zero probabilities for unseen words.09ClusteringMethodShape assumptionNeeds k?Notesk-meansSpherical, similar sizeYesFast; sensitive to scaling and initDBSCANDensity-connectedNoFinds noise; tune eps and min samplesHierarchicalAny (by linkage)NoDendrogram; O(n²) memoryGMMEllipticalYesSoft assignments, probabilisticChoosing kElbow on inertia, silhouette score, or downstream usefulness.10Dimensionality ReductionPCALinear projection onto directions of maximum variance; standardize first.Explained varianceKeep enough components to retain about 90–95% of variance.t-SNE / UMAPNonlinear embeddings for visualization; distances between clusters are not meaningful.AutoencodersLearn a compressed representation with a neural network.11Feature EngineeringNumericStandardize, log-transform skewed values, clip extreme outliers.CategoricalOne-hot for low cardinality; target or hash encoding for high cardinality.DatesExtract day of week, month, recency, and time since an event.TextTF-IDF for classic models; embeddings for semantic tasks.InteractionsRatios and products often beat raw features for linear models.Missing valuesImpute and add an "is missing" indicator when absence is informative.12Class ImbalanceUse the right metricPR-AUC, recall at fixed precision, or cost-weighted error, not accuracy.Class weightsPenalize minority errors more inside the loss.ResamplingUndersample the majority or oversample the minority (SMOTE) on train only.Threshold tuningMove the decision threshold instead of retraining.Resample after splittingOversampling before the train/validation split copies minority rows into both sets and inflates scores. 13Cross-Validationk-foldRotate k validation folds; average the scores.Stratified k-foldPreserves class ratios per fold; default for classification.Group k-foldKeeps all rows of a user or entity in one fold.Time series splitTrain on the past, validate on the future, never shuffle.Nested CVInner loop tunes, outer loop estimates the tuned model's performance.14Data LeakageTarget leakageA feature is computed using information from after the prediction time.Preprocessing leakageScalers or imputers fit on the full dataset instead of train only.Duplicate leakageThe same entity appears in both train and test.Temporal leakageRandom splits on time-ordered data.Too good to be true usually isA sudden jump in validation score is a leakage signal. Inspect the top features first. 15Model Selection at a GlanceSituationGood first choiceSmall tabular dataLogistic / linear regression, random forestLarge tabular dataGradient boosting (LightGBM, XGBoost)Text classificationTF-IDF + linear model, then fine-tuned transformerImages, audioPretrained CNN or transformerNeed explanationsLinear models, shallow trees, SHAP16Hyperparameter IntuitionLearning rateToo high diverges, too low is slow; tune on a log scale.Tree depthDeeper fits interactions but overfits; boosting prefers shallow trees.n_estimatorsMore trees help random forests; boosting needs early stopping.Regularization λ / CTrade training fit for generalization.SearchRandom or Bayesian search beats grid search in high dimensions.17Final Checklist1.The target and prediction time are defined precisely.2.The split matches how the model will be used (time, group).3.A simple baseline is recorded.4.All preprocessing is fit on the training fold only.5.The metric reflects business costs, not just accuracy.6.Results include uncertainty (CV spread or confidence interval).