ML Fundamentals
5 interview questions in this topic. Expand any question to read the full answer.
“Explain the bias-variance tradeoff”
Answer
A model's expected error on unseen data breaks into three pieces:
Expected error = Bias² + Variance + Irreducible noise
Bias is error from wrong assumptions — a model too simple to capture the real pattern, so it misses even on the training set (underfitting). Variance is error from sensitivity — a model so flexible it fits the noise in the training set, so it looks great on seen data and falls apart on new data (overfitting). Irreducible noise is randomness in the data itself and sets the floor on your error.
The tradeoff is the tug-of-war between the first two. As you add model complexity, bias falls but variance rises, so total error traces a U-shape: too simple on the left, too complex on the right, and a minimum in the middle — the model you actually want.
This is also a practical diagnostic. If training error is already high, you're bias-bound and need more capacity or better features. If training error is low but test error is high, you're variance-bound and need regularization or more data.
💡 Note Think of throwing darts. High bias is aiming at the wrong spot — the darts cluster tightly but off-center. High variance is a shaky hand — you're aimed at the bullseye but the darts scatter. You want a steady hand aimed at the center (low bias, low variance), but every adjustment trades a little of one for the other.
Related Questions
“How does gradient descent work?”
Answer
Training a model means finding the parameters that minimize a loss function J(θ) — a measure of how wrong the model is. For anything beyond the simplest models there's no closed-form solution, so you search for the minimum step by step. That's gradient descent.
The gradient ∇J(θ) points in the direction of steepest increase of the loss, so to decrease the loss you step the opposite way. That gives the update rule:
θ := θ − η · ∇J(θ)
New parameters equal old parameters minus a small step in the gradient's direction. The η (eta) is the learning rate, the size of each step — and it's the hyperparameter that makes or breaks training. Too small and training crawls; too large and you overshoot the minimum and can diverge.
There are three common variants: batch gradient descent uses the whole dataset per step (accurate but slow), stochastic (SGD) uses one example (noisy but cheap), and mini-batch uses a small batch of 32–256 — the practical default. One more distinction interviewers probe: backpropagation computes the gradient efficiently in a neural network, while gradient descent uses that gradient to update the weights. They're partners, not the same thing.
💡 Note You're on a mountain in thick fog and want to reach the valley. You can't see the bottom, but you can feel which way the ground slopes under your feet, so you step downhill, feel the slope again, and step again. The learning rate is your stride length: tiny steps are safe but slow, giant leaps might overshoot the valley entirely.
Related Questions
“What is overfitting and how do you prevent it?”
Answer
Every dataset is signal plus noise. Signal is the real relationship you want to capture; noise is the random junk specific to this sample. Overfitting is when a model is flexible enough to fit the noise as if it were signal, so it describes this dataset perfectly and the real world badly.
The clearest symptom is the gap between training and validation error. A model that gets 99% on training and 72% on validation isn't unlucky — it's overfit. It usually happens for one of three reasons: too much capacity for the amount of data, training too long, or leaky/too-few features.
You prevent it two ways — shrink the model or grow the data:
- Shrink the model: regularization (L1/L2) to penalize large weights, early stopping before it starts memorizing, dropout in neural nets, or simply a simpler architecture with fewer features.
- Grow the data: more training examples (the most reliable fix, since noise averages out), or data augmentation when collecting more isn't practical.
Underneath it all, use cross-validation for an honest read on generalization, so you're tuning against real performance instead of a lucky split.
💡 Note Overfitting is the student who memorizes the exact answers to last year's exam. They ace the practice test and freeze on the real one, because they learned the answer key instead of the subject. You want a model that learned the subject.
Related Questions
“What is regularization?”
Answer
Left alone, an optimizer will do whatever drives training loss down, which often means growing huge, finely-tuned weights that contort the model to hit every training point — including the noise. Regularization changes the objective so complexity has a cost:
Total loss = Data loss + λ · Penalty(weights)
That penalty pulls weights toward zero, keeping the learned function smoother and less able to memorize noise. The two you'll be asked about:
- L2 (ridge) penalizes the sum of squared weights. It shrinks all weights smoothly toward zero without forcing any to disappear — a great default when most features contribute a little.
- L1 (lasso) penalizes the sum of absolute weights. Its geometry drives some weights exactly to zero, so it performs feature selection and yields a sparse, interpretable model.
The knob is λ (lambda), the regularization strength: at λ = 0 you're back to no regularization; push it too high and the model underfits. You tune it with cross-validation. Tie it back to bias-variance and it clicks — regularization deliberately adds a little bias to buy a large reduction in variance. And it's broader than weight penalties: dropout, early stopping, and data augmentation are regularizers too, just applied through training rather than the loss.
💡 Note Regularization is a tax on complexity. The model can still buy extra flexibility if it genuinely improves the fit, but now it has to pay for it — so it only splurges when the data really justifies it.
Related Questions
“Why do we use cross-validation?”
Answer
A single train-test split is one sample of your model's performance, and a noisy one. Get an easy test set by chance and the model looks great; get a hard one and it looks weak. On smaller datasets that swing can be big enough to make you pick the wrong model.
Cross-validation reduces that luck. The workhorse is k-fold:
- Split the data into k equal folds (k = 5 or 10 is typical).
- Train on k−1 folds, evaluate on the held-out fold.
- Rotate so every fold gets a turn as the test set.
- Average the k scores for your estimate, and look at their spread for a sense of stability.
Because every example is used for training in most rounds and for testing in exactly one, you get an honest estimate out of all your data. That averaged score has lower variance than any single split, so when you use it to compare models or tune hyperparameters, you're choosing on real signal rather than a lucky partition.
A few variants signal depth: stratified k-fold preserves class balance (use it for imbalanced classification), time-series split trains on the past and tests on the future without shuffling, and group k-fold keeps all rows from one entity in the same fold. The cost is compute — k-fold trains the model k times — which is the tradeoff you accept for an estimate you can trust.
💡 Note Judging a competition on a single round is a coin toss. Cross-validation is scoring every contestant across multiple rounds and averaging — one bad round doesn't decide the winner, and you trust the ranking more.