AdaBoost
Misclassified points get louder — exponential loss as a reweighting story.
AdaBoost builds a strong model out of many weak ones. It starts with a "weak learner"—typically a decision stump that makes a single split. It gets a lot wrong, but it only needs to be slightly better than guessing.
Reweighting the Data
After the first stump evaluates the data, the points it got right are down-weighted. The points it misclassified are given heavier weights.
Focus on the Hard Examples
The next stump in the sequence is forced to focus on the heavy, misclassified points. It builds its rule to fix the previous stump's mistakes, ignoring the easy points. This process repeats sequentially.
The Weighted Ensemble
To make a final prediction, every stump gets a vote. However, stumps that made fewer mistakes overall get heavier votes than inaccurate stumps.
Where It Breaks
Because AdaBoost aggressively up-weights errors, a single mislabeled point or extreme outlier can destroy the model. The algorithm will fixate on the impossible point, inflating its weight until the rest of the data is ignored.
The Quick Version
- Weak Learner: Uses simple models that are just better than random.
- Reweighting: Incorrect predictions get heavier weights for the next round.
- Sequential: Each model tries to fix the mistakes of the previous one.
- Aggregation: The final prediction is a weighted vote.
- Failure: Highly sensitive to noisy data and outliers.