Exploratory Data Analysis

Why Does This Exist?

A used-car dealer builds a model that predicts what each car will sell for. In testing it is superb. In its first live week it loses money on nearly every deal, because for three rows in ten the nightly job had copied list_price into sale_price whenever the real figure arrived late. The model learned to hand back the list price. An echo, not a prediction.

Nobody had plotted sale_price against list_price. Every single-column summary was reassuring: plausible means, no blanks, no absurd extremes. The fault was never inside a column. It lived in a pair, and no summary table looks at pairs.

That is the job this page describes. Not plotting until you feel better, but a fixed interrogation in a fixed order whose output is written down: which columns are unusable, which need a transform, what the split must respect, and which rows need a human to go and ask about them. If you cannot produce those four lists, you looked at your data but you did not question it.

This page covers the questioning itself. Fixing what you find belongs to its neighbours: filling gaps is missing data, and the discipline of keeping test rows out of every decision is data leakage.

The one idea you need first. A distribution is the shape of one column: where its values pile up, which tail runs long, and which single values carry a suspicious share of the rows. A relationship is how two columns move together, and it can be strong where both distributions look innocent. The first pass below reads shapes; the second reads relationships.

Think of It Like This

How It Actually Works

Three passes over the same table, in this order, because each pass is only readable once the previous one is done.

Pass one: one column at a time

Draw a histogram per column. A summary table is not a substitute, because two numbers cannot carry a shape. Read whether the mass sits in one hump or two, which tail runs long, and whether the range is physically possible. Watch for spikes on a single value: 0, -1 and 999 are usually sentinels, blanks wearing the clothes of measurements, and they must become nulls before anything is fitted.

Each finding becomes a written entry. Sentinels convert to null, a long right tail wants a log transform for a linear model and nothing at all for a tree, and the blank fraction per column feeds missing data.

Pass two: two columns at a time

Numeric against numeric gets a scatter, numeric against categorical gets a box plot per category. Look for direction and curvature, since those want different models. Look for points sitting on a line they have no business sitting on: an exact functional relationship means one column was computed from the other. And plot every candidate against the target, because a clean relationship there is a suspect more often than a gift.

Pass three: many columns at once

Draw the correlation matrix as a heatmap. Blocks of mutually correlated columns mean you are carrying the same information several times, which distorts linear coefficients and changes what dimensionality reduction keeps. A relationship that flips sign once you condition on a group only shows up here. Pass two says a relationship exists. Pass three says whether you believed the right reason for it.

Split before the first plot

Split first, then explore the training rows and nothing else. People wave this off because no code is involved. But you are the fitting step: you see a long tail in the test rows, you pick a log transform, and that decision is a parameter set from held-out data. Nothing recorded it, so no audit will ever find it. The channel was a person.

Worked example

Eight cars, list price and sale price in hundreds:

1list_price  sale_price2   240         2283   300         2854   180         1715   455         4556   280         2667   520         5208   200         1909   380         380

Pass one on sale_price reports mean 311.875, range 171 to 520, no blanks, nothing bimodal. list_price reports mean 319.375, range 180 to 520. A summary table hands both columns a clean bill.

Pass two divides one by the other. The ratio is 0.95 five times and exactly 1.00 three times, so a scatter puts five points on a discount line and three exactly on $y = x$. Markets do not do that. The correlation is 0.999, which is also what a genuinely useful feature looks like. The tell is the shape, not the number.

Code

The same eight rows twice. Pass one clears them, pass two does not.

1import numpy as np2
3list_price = np.array([240.0, 300.0, 180.0, 455.0, 280.0, 520.0, 200.0, 380.0])4sale_price = np.array([228.0, 285.0, 171.0, 455.0, 266.0, 520.0, 190.0, 380.0])5
6
7def univariate(col: np.ndarray) -> tuple[float, float, float]:8    return float(col.mean()), float(col.min()), float(col.max())9
10
11def distinct_ratios(a: np.ndarray, b: np.ndarray) -> list[float]:12    return np.unique(np.round(b / a, 3)).tolist()13
14
15print(univariate(sale_price))  # -> (311.875, 171.0, 520.0)16print(round(float(np.corrcoef(list_price, sale_price)[0, 1]), 3))  # -> 0.99917print(distinct_ratios(list_price, sale_price))  # -> [0.95, 1.0]

Two distinct ratios where a market hands you eight.

Watch Out For

The Quick Version

• The output is a written list of decisions, not a folder of plots.
• Three passes, in order: one column, two columns, many columns.
• Pass one is the histogram. Hunt shape, range, sentinel spikes, near-constant columns.
• Pass two is the scatter. An exact relationship between two columns means one was computed from the other.
• Pass three is the correlation matrix, for duplicated information and sign flips that appear only under grouping.
• Split before the first plot. Picking a transform after seeing test rows is leakage with no code in it.
• Identical means, variances and correlations sit over four unrecognisably different shapes.
• A feature that tracks the target too cleanly is a suspect, not a win.

What to Read Next

• Feature Construction
• Feature Engineering