GBDT Libraries Disagree on Categorical Columns

Why Does This Exist?

For years, the standard advice for tabular data was to one-hot encode categorical variables before feeding them to a gradient boosted decision tree (GBDT). But one-hot encoding creates massive, sparse matrices that slow down training and destroy the tree's ability to make deep splits on high-cardinality features.

Eventually, the big three libraries—XGBoost, LightGBM, and CatBoost—all added "native" categorical support. You just pass in a list of column indices and the algorithm figures it out. Developers assumed this meant the three libraries had converged on a standard approach. They hadn't.

Think of It Like This

Imagine asking three different chefs to make a pizza. They all hand you a pizza, but one used a wood-fired oven, one used a cast-iron skillet, and one deep-fried it. The interface (pizza) is identical, but the internal chemistry is completely different.

How It Actually Works

When you declare a column as categorical, each library takes a radically different mathematical path:

• LightGBM uses Fisher's optimal partition method. It calculates the average target value for each category, sorts the categories by that average, and then finds the best single split point on that sorted list. It is blazing fast but prone to severe overfitting on high-cardinality columns.
• CatBoost uses ordered target statistics. It essentially replaces the category label with the expected target value, but it does so sequentially based on a random permutation of the data to prevent data leakage. It is incredibly robust to overfitting but slower to train.
• XGBoost recently introduced one-hot based splits natively inside the tree for low cardinality, and partition-based splits for high cardinality. It tries to hybridize the approach, but the implementation is highly dependent on which tree method (hist vs exact) you are using.

Watch Out For

If you migrate from LightGBM to XGBoost and simply pass the categorical_feature flag, your model's behavior will change drastically. The way they handle unseen categories during inference is also different. LightGBM will map unseen categories to the branch with the most data; CatBoost will treat them as a prior.

(Correct as of August 2026, comparing XGBoost 2.1, LightGBM 4.4, and CatBoost 1.2).

The Quick Version

Just because an algorithm accepts a categorical flag does not mean it treats the data the same way. The divergence between LightGBM, CatBoost, and XGBoost's categorical handling will drastically shift your feature importance and overfitting risk.

What to Read Next

Check the updated gradient-boosting-machines concept page to understand the math behind these splits.