Random Forests
Formalized Random Forests, an ensemble learning method that combines bagging and random feature selection to build highly robust and accurate predictive models.
The Problem
Decision trees are highly interpretable and easy to build, but they suffer from high variance: they are incredibly prone to overfitting. A tree will often memorize the exact training data it was given, making its predictions erratic and unreliable on new, unseen data.
The Idea
If one tree is prone to overfitting and high variance, what if we train hundreds of slightly different trees and average their predictions? This concept, called 'Bagging' (Bootstrap Aggregating), was already known. Breiman's stroke of genius was adding a second layer of randomness: instead of letting every tree look at every feature to make a split, force each tree to choose from a small, random subset of features.
How It Works
- Bootstrapping: Create different training datasets by sampling the original data with replacement.
- Tree Building with Random Subspaces: Train a decision tree on each dataset. However, at each node in the tree, instead of searching all features for the best split, randomly select a small subset of features (e.g., the square root of the total features) and only split based on those.
- Aggregation: Let all the trees grow completely unpruned. For classification, the forest votes (majority wins). For regression, it averages the predictions.
By forcing the trees to use different subsets of data and different subsets of features, the forest guarantees that the trees are de-correlated. When you average de-correlated models, the variance plummets without increasing the bias.
Why It Mattered
Random Forests became the absolute gold standard for tabular data for over a decade. It is incredibly easy to use: it requires almost no hyperparameter tuning, it is highly resistant to overfitting, and it naturally provides a measure of 'feature importance'.
What Came After
Random Forests remain heavily used today in industry, finance, and bioinformatics for tabular data. While Gradient Boosting (like XGBoost) eventually surpassed it in pure predictive accuracy, Random Forests are still preferred when simplicity, speed of training, and robustness without tuning are the priorities.