Core ML

Tabular Deep Learning

Deep learning dominates images, text, and audio. But if you throw a standard neural network at an Excel spreadsheet, it will usually lose to a 10-year-old tree-based model. Tabular Deep Learning tries to fix this.

Tree-based models build hard, axis-aligned boundaries that perfectly partition tabular data. Standard Neural Networks build smooth, curved boundaries that struggle to represent sharp categorical shifts.

Why Does This Exist?

If you want to classify an image, translate text, or generate a voice, Deep Learning is the undisputed king. But if you go to Kaggle and look at competitions involving tabular data (rows and columns, like a SQL database or Excel spreadsheet), Deep Learning almost never wins. The winners use Gradient Boosted Decision Trees (GBDTs) like XGBoost, LightGBM, or CatBoost.

Why does the most advanced AI technology in the world fail on the most common data format in business?

Tabular Deep Learning is the sub-field of ML research dedicated to answering that question, and building novel neural network architectures designed specifically to beat XGBoost on its home turf.

Think of It Like This

Images and text are Homogeneous. Every pixel in an image is the same unit (color intensity). A CNN can slide a window across an image because a pixel at the top left is fundamentally the same type of thing as a pixel at the bottom right.

Tabular data is Heterogeneous. Column 1 is "Age" (a continuous integer). Column 2 is "Income" (a massive skewed float). Column 3 is "City" (a categorical string). Column 4 is "Is_Subscribed" (boolean). Trying to use a standard Neural Network on this is like trying to blend a rock, a feather, and a cup of water in a blender. They don't mix well mathematically. Decision Trees, on the other hand, don't try to mix them. They just draw strict boundaries: "If Age > 30 and City == 'NY'".

Why Deep Learning Struggles

Researchers (like Grinsztajn et al., 2022) have identified three main reasons why standard Multi-Layer Perceptrons (MLPs) and ResNets fail on tabular data:

Lack of Locality / Inductive Bias: In an image, two adjacent pixels are highly related. In a table, Column 2 and Column 3 might have absolutely nothing to do with each other. Neural networks struggle when there is no spatial structure to exploit.
Smoothness vs. Irregularity: Neural networks naturally learn smooth, curved decision boundaries. Tabular data often requires sharp, axis-aligned step functions (e.g., if a user crosses the age of 18, their legal status changes instantly). Trees naturally build step functions. NNs struggle to simulate them.
Uninformative Features: Tabular datasets often contain hundreds of useless columns. Decision trees simply ignore them (by never splitting on them). Standard NNs feed every column into the first linear layer, letting the noise ruin the gradient.

How Tabular Deep Learning Fixes It

To beat trees, researchers have designed specialized neural architectures for tables.

1. TabNet (Google, 2019)

TabNet uses a mechanism called Sequential Attention to mimic the behavior of a decision tree. At each step in the network, it calculates a mask that forces the network to completely ignore useless columns, focusing only on the 2 or 3 most important columns for that specific row. This provides both high accuracy and built-in explainability.

2. FT-Transformer (Yandex, 2021)

The Feature Tokenizer Transformer takes the architecture behind ChatGPT and applies it to tables. It takes every single cell in the table (whether it's a number like 42 or a category like 'Blue') and projects it into a high-dimensional embedding vector (token). It then passes these tokens through standard Transformer self-attention layers. This allows the network to learn incredibly complex interactions between specific columns.

Show Me the Code

You can use the pytorch-tabnet library to easily train a TabNet model on tabular data, using an API that looks exactly like scikit-learn.

1import pandas as pd2from pytorch_tabnet.tab_model import TabNetClassifier3from sklearn.model_selection import train_test_split4from sklearn.metrics import accuracy_score5
6# X: A standard pandas DataFrame with mixed numerical and categorical data7# y: Binary target (0 or 1)8
9X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)10
11# 1. Initialize TabNet12# It mimics a tree ensemble but trains via gradient descent13clf = TabNetClassifier(14    n_d=8, n_a=8, # Width of the decision and attention steps15    n_steps=3,    # Number of "tree splits" it simulates16    gamma=1.3,17    verbose=018)19
20# 2. Fit the model21# TabNet natively handles early stopping based on validation loss22clf.fit(23    X_train=X_train.values, y_train=y_train.values,24    eval_set=[(X_test.values, y_test.values)],25    patience=10, max_epochs=10026)27
28# 3. Predict29preds = clf.predict(X_test.values)30print(f"TabNet Accuracy: {accuracy_score(y_test, preds):.3f}")31
32# 4. Global Explainability33# TabNet automatically tells you which columns were actually used34feature_importances = clf.feature_importances_35print("Feature Importances:", feature_importances)

Watch Out For

Over-engineering. Tabular Deep Learning models are notoriously difficult to tune. They require massive datasets, GPU acceleration, and careful hyperparameter searches. Meanwhile, you can pip install XGBoost, run it on a CPU with default hyperparameters, and get 99% of the performance in 5 seconds. For most business applications, GBDTs are still the undisputed champion.

The Quick Version

Deep Learning struggles with tabular data because tables are heterogeneous (mixed data types), contain uninformative features, and require sharp decision boundaries.
Gradient Boosted Decision Trees (XGBoost, LightGBM) natively solve these problems and are the industry standard for tabular data.
Tabular Deep Learning architectures (like TabNet and FT-Transformer) attempt to bridge the gap by mimicking tree-like feature selection or using self-attention to process columns as tokens.

Related concepts

Tabular Foundation Models