Random Forests

Why Does This Exist?

Averaging only helps when the things you average disagree, and one dominant column is what usually stops them.

Here is the case carried down this page. You run a chain of gyms and want to know who renews next month. Nine thousand members, thirty columns each: plan tier, joining month, distance from home, classes booked, and the one column that predicts anything, visits in the last thirty days.

Bagging gets part of the way. Resample the rows, fit a deep tree on each, average the votes. Then the arithmetic bites: averaging $B$ correlated models leaves variance $\rho\sigma^2 + (1-\rho)\sigma^2/B$, where $\sigma^2$ is one tree variance and $\rho$ is their correlation. More trees shrink the second term. The first term is a floor.

Gym data pushes that floor high. The visits column sits in every resample, wins the root split in every tree by a wide margin, and the level below usually agrees too. Two hundred trees rooted on visits at nearly the same threshold are not two hundred opinions. They are one opinion with rounding noise.

The fix is to take the choice away from them.

Think of It Like This

How It Actually Works

A second dice roll, at every split

A forest keeps the bootstrap of the rows and adds one rule inside the tree. At every split, draw $m$ of the $p$ columns at random and consider only those. Defaults are $m = \sqrt{p}$ for classification and $m = p/3$ for regression, so thirty columns means five candidates. Not five for the tree, five for that split, redrawn at the next one.

Visits is therefore on offer at about one split in six. The rest of the time the tree builds from commute distance and class bookings, which works because the rows carry signal without their best column.

This is deliberate handicapping. Each tree comes out weaker than a bagged tree, so $\sigma^2$ rises, but $\rho$ falls faster, and $\rho$ is the term compute cannot touch. That trade is the whole design. The bias-variance tradeoff names the two terms being traded.

Grow them deep, and stop counting

Trees in a forest are grown deep and unpruned on purpose. A lone tree that deep memorises individuals, which is what pruning prevents. Inside a forest, averaging is the variance treatment, so pruning would only add bias the ensemble was never going to suffer.

More trees never overfit. Adding them drives the second variance term toward zero and then stops, so the curve flattens rather than turning back up. Tree count is a budget, not a dial. Go to 500, watch it flatten, stop paying. The dials that change the model are $m$ and the minimum rows per leaf. Against gradient boosting, a tuned boosted model usually wins a tabular benchmark, and a forest is much harder to get wrong.

Validation for free

Each resample leaves about 36.8 percent of rows behind, and those rows are out of bag for the tree fitted on it. Every member has roughly a third of the forest for which it was out of bag, so averaging those predictions per member gives validation with nothing reserved and nothing extra fitted.

Two limits. At 50 trees a row is scored by around 18, a weaker ensemble than the one shipped, so the estimate reads pessimistic. And the trick assumes independent rows, which the pitfall below breaks. For the resampling math behind the 36.8 percent, see bagging.

Worked mini example: one root against eighteen

Take forty trees on the gym data. Offer all thirty columns at every split and the visits stand-in wins the root forty times out of forty, so the distinct root count is 1. Restrict each split to five columns and the same forty trees root on eighteen different columns. Same rows, same depth, same code path. The only change is the subset size, and the root diversity jumps from 1 to 18. That jump is the correlation drop, visible before a tree has finished growing.

Code

One split, scored twice: all thirty columns on offer, then five. Count the distinct root winners across forty trees.

1import numpy as np2
3rng = np.random.default_rng(0)4x = rng.normal(size=(600, 30))5renews = (1.8 * x[:, 0] + 0.6 * x[:, 1] + rng.normal(0.0, 0.5, 600) > 0).astype(int)6
7def root_split(rows: np.ndarray, cols: np.ndarray) -> int:8    y, gains = renews[rows], []9    for f in cols:                                   # only the drawn columns are even looked at10        lo = x[rows, f] <= np.median(x[rows, f])      # one candidate cut per column, at its median11        gains.append(y.var() - lo.mean() * y[lo].var() - (1 - lo.mean()) * y[~lo].var())12    return int(cols[int(np.argmax(gains))])13
14def distinct_roots(m: int, trees: int = 40) -> int:15    picks = [root_split(rng.integers(0, 600, 600), rng.choice(30, m, replace=False)) for _ in range(trees)]16    return len(set(picks))17
18print(f"all 30 columns offered: {distinct_roots(30)} distinct root split | 5 of 30 offered: {distinct_roots(5)}")19# -> all 30 columns offered: 1 distinct root split | 5 of 30 offered: 18

Column 0 is the visits stand-in. Offered every time, it wins the root forty times out of forty. Restrict each split to five columns and the same forty trees root on eighteen different ones. Change the seed and the eighteen moves a little, but the one against many gap stays.

Watch Out For

The Quick Version

• Random forests add a second dice roll to bagging: at each split only a random $m$ of $p$ columns are considered.
• Defaults are $m$ equals square root of $p$ for classification and $p$ over 3 for regression, redrawn at every split.
• With thirty columns and five offered, visits appears at about one split in six, so forty trees go from one distinct root to eighteen.
• Grow trees deep and unpruned because averaging handles variance, and more trees flatten cost rather than overfit.
• About 36.8 percent of rows sit out of bag per tree, which gives free validation that turns pessimistic with few trees and optimistic with grouped rows.
• Impurity importance is not a finding because cardinality inflates it and twins split credit, so check with permutation on held out data.

What to Read Next

• Bagging
• Decision Trees
• Gradient Boosting