Supervised Ensembles: Random Forests & Boosting

What you are looking at

A dataset forming an XOR pattern—two classes arranged diagonally across four clusters. A single, simple decision tree (a "weak learner") is completely incapable of drawing a boundary that separates this data accurately. It can only make one horizontal or vertical split, leaving half the points misclassified.

Ensemble methods combine multiple weak learners into a single, highly accurate predictor. The two most common strategies are Bagging (Random Forests) and Boosting (AdaBoost).

Random Forests (Bagging)

Random Forests train dozens of independent trees in parallel. To ensure these trees are diverse, each one is trained on a random sample of the training data (a bootstrap sample). When making a prediction, the ensemble simply takes a majority vote from all its constituent trees.

• Diversity is key: If all trees were the same, their vote would not improve anything. The random sampling ensures that each tree focuses on slightly different aspects of the data.
• Robustness: Because individual trees overfit to different noise in their respective samples, averaging their predictions cancels out the noise, resulting in a smooth, robust decision boundary.

Boosting

While Random Forests train trees independently, Boosting trains them sequentially. Each new tree in the ensemble focuses specifically on correcting the mistakes made by the previous trees.

1. Start with a weak model (like a decision stump).
2. Evaluate its performance. Points that were misclassified are assigned a higher weight (represented visually by larger point sizes).
3. Train the next model, forcing it to pay more attention to the highly weighted points.
4. Repeat this process, continually shifting focus to the hardest examples.

The final prediction is a weighted sum of all the trees, where trees with lower error are given more influence.

Break it on purpose

Ensembles are powerful, but they handle bad data differently. Random Forests are highly robust against outliers because bagging averages out the noise. Boosting, on the other hand, is notoriously sensitive to outliers.

The vulnerability of Boosting

Because Boosting aggressively increases the weight of misclassified points round after round, a single extreme outlier can hijack the entire training process. The ensemble will warp its boundary and sacrifice general accuracy just to classify that one impossible point correctly.

Try it in the simulation: hover over a point to see how the trees in the Random Forest vote. Notice how the Random Forest boundary remains relatively stable. Then, switch to Boosting and watch the sample weights grow exponentially for the points the stumps struggle to classify. If one point was completely out of bounds, Boosting would dedicate all its trees to capturing it.