XGBoost, LightGBM And CatBoost Disagree On Categories

Teams treat XGBoost, LightGBM and CatBoost as interchangeable — same objective, same tree-based mechanism, pick whichever one your infra already runs. That's true for numeric features. It stops being true the moment a column is categorical, because the three libraries answer "what do I do with this string column" in three different ways, and switching between them changes what the model learns, not just how fast it trains.

Three answers to the same question

Gradient-boosted tree libraries all bin continuous features into histograms and grow trees against gradient and curvature sums per bin — that part is shared ground. Categorical handling is where they diverge.

XGBoost added native categorical support starting at version 1.5, letting gpu_hist and later approx split directly on category membership rather than requiring you to one-hot or ordinally encode first — documented as experimental at introduction and matured across subsequent releases. Content was rephrased for compliance with licensing restrictions. Absent that native path, the historical default was one-hot encoding, which is exactly the failure mode described on the gradient-boosted tree libraries page: thousands of near-empty columns for a high-cardinality feature.

LightGBM takes a different route: it groups category values by a Fisher-partition method to find near-optimal splits without one-hot expansion, treating the column as categorical throughout rather than pre-encoding it into numeric bins.

CatBoost goes further, with ordered target statistics as its default categorical encoding — the scheme covered on the gradient-boosted tree libraries page, where each row's encoding is computed only from rows earlier in a random permutation, specifically to prevent target leakage through the encoded column. Comparative benchmarks note it as the library that performs best when categorical variables are handled properly, precisely because that handling is load-bearing rather than optional. Content was rephrased for compliance with licensing restrictions.

Why this is a modelling decision, not a config flag

Three consequences follow directly from the three mechanisms. First, leakage risk differs by library and by your own preprocessing. If you hand XGBoost or LightGBM a column you target-encoded yourself using the whole training set, you've built the leak the gradient-boosted tree libraries page warns about — CatBoost's ordered scheme exists specifically to avoid that failure by construction, but only if you let it do the encoding rather than pre-encoding for it.

Second, feature importance rankings are not comparable across libraries on the same categorical column, because the column is represented differently under the hood — a native categorical split, a Fisher-grouped bin, and an ordered-statistic numeric feature are three different objects competing for the same splits.

Third, swapping libraries without re-auditing your encoding pipeline is a silent behavior change. A pipeline built around CatBoost's expectation that raw categorical columns are safe to hand over directly will, if ported to XGBoost without adjustment, either fail on unsupported dtypes or silently fall back to a numeric interpretation that treats an unordered category as ordered.

The practical takeaway

Before switching gradient-boosting libraries on a project with meaningful categorical features, re-derive your encoding strategy rather than assuming the swap is a drop-in. Check which library actually consumed the encoding you intended: pass raw categorical columns to CatBoost, use native categorical support in XGBoost or LightGBM where cardinality warrants it, and never hand a manually target-encoded column to a library that also encodes categoricals itself — you'll be doing the leak-prone version twice.