Data Concepts

Data Validation and Contracts

The good failure breaks the pipeline, so you find out immediately. The expensive one keeps everything running and quietly changes what the numbers mean.

A batch passes the schema, value and cross-field gates and stops at the distribution gate, which is the only layer that notices a column whose unit changed

Why Does This Exist?

An upstream team changes amount from cents to pounds. Reasonable change, mentioned in their standup, not in yours.

Your pipeline doesn't notice. The column is still an integer, still non-null, still positive, still under any ceiling anybody set. Every check passes. Every model retrains. Every threshold that was tuned against amounts in the thousands now sits a hundred times too high, so the fraud model stops flagging anything and the revenue dashboard is wrong by two orders of magnitude for five weeks.

Here's the asymmetry worth internalising. A change that breaks the pipeline is the good case — you find out in ten minutes, from a stack trace, and fix it. The expensive case keeps everything running and silently changes what the numbers mean.

This is the cheapest production failure to prevent, and it's skipped more than anything else in this band, because it's the only work here that produces nothing visible when it succeeds.

Think of It Like This

The goods-in bay, and the one check nobody wrote down

A kitchen has a clipboard at the delivery bay. Weight matches the invoice, temperature under 5°C, box labelled with what's inside. Fail a line and the crate goes back with a note.

One week the supplier switches from 2kg bags of flour to 1kg bags and doubles the count. Every line on the clipboard passes: the weight is right, the temperature is right, the label says flour. And every recipe in the kitchen that says "one bag" is now half strength.

Nothing was broken. Nothing was mislabelled. The clipboard checked what somebody thought to write down in 2019, and "a bag means 2kg" was so obvious that nobody wrote it.

That unwritten assumption is what a contract is for. Not the things you'd notice — the things you rely on without saying so.

How It Actually Works

Four layers of assertion

Schema. Which columns exist, their types, their nullability, their order if anything depends on it. Note the asymmetry: an added column is usually harmless, while a renamed or retyped one is not, so a strict "no unknown columns" rule generates friction without buying much.

Values. Ranges (age between 0 and 120), allowed sets (currency in three values), formats (a postcode regex), uniqueness (order_id), referential integrity (every customer_id resolves). Cheap, and it catches the majority of genuinely broken rows.

Distributions. Mean, quantiles, null rate, category frequencies, distinct count — each compared against a reference window. This is the layer that catches the unit change, and it's the layer people skip, because it's the only one that needs a reference and a threshold rather than a rule you can state from first principles. It's also the earliest warning you get for drift.

Cross-field and cross-batch. ended_at after started_at, line items summing to the order total, timestamps monotone, and row count within a sensible band of yesterday's. A batch arriving at 3% of normal volume is a broken export, and no single-column check sees it.

Where the gate goes

At pipeline boundaries — wherever data changes hands between teams or systems, because that's where assumptions diverge. And at both entry points to the model: the training path and the serving path. Validating only at training means production can feed the model anything, which is half of how skew survives.

Fail closed, fail open, and being honest about it

Blocking a batch is the strong move and it has a cost. A fail-closed distribution check on a seasonal column stops the pipeline every Christmas, and the fix people actually apply at 2am is to comment it out.

So split by layer. Schema and value checks block — a string in an integer column or a negative age is unambiguously wrong, and processing it makes things worse. Distribution checks warn by default, with a severity ladder, and only block on a threshold so extreme that no legitimate business event reaches it. A hundredfold shift in a mean is that extreme. A 40% December uplift isn't.

A contract has two parties

The layer nobody writes about. A validation suite you own alone is a tripwire: it tells you the producer changed something, after they changed it.

A contract is the version-controlled statement that a specific shape is depended on, sitting next to the schema, in the producer's repository, reviewed when they change it. That converts your 3am alert into their failing CI job. It's a social artefact more than a technical one, and it's the difference between detecting breakage and preventing it. A wiki page is not this.

How it ties into the rest of the band

Validation is where cleaning rules stop being notebook cells and become permanent. It's the earliest place drift is visible. It's what makes skew detectable, by asserting the same shape on both paths. And it's only reproducible if you can say which data you asserted against, which is data versioning.

Worked example

Five reference rows of amount in cents: 1200, 3800, 4250, 5100, 9900. Mean 4850.

Next week the same column arrives as 12, 38, 43, 51, 99 — pounds now, rounded to integers.

Walk the layers. Schema: still int64, still non-null. Passes. Values: every row is above 1 and below a million. Passes. Cross-field: nothing contradicts, row count identical. Passes.

Distribution: the mean went from 4850 to 48, a ratio of 99.8. That's the only assertion in the four layers that has anything to say, and it says it loudly enough that no seasonal event could produce it.

The near-100 rather than exactly-100 is the rounding. Which is a small nice thing: the ratio tells you it was a unit change and roughly which one.

Show Me the Code

1import numpy as np2
3reference: np.ndarray = np.array([1200, 3800, 4250, 5100, 9900], dtype=np.int64)  # cents4batch: np.ndarray = np.array([12, 38, 43, 51, 99], dtype=np.int64)                # silently pounds5
6def schema_ok(x: np.ndarray) -> bool:7    """Right type, no nulls. The layer most pipelines stop at."""8    return x.dtype == np.int649
10def range_ok(x: np.ndarray, lo: int = 1, hi: int = 1_000_000) -> bool:11    return bool(x.min() >= lo and x.max() <= hi)12
13def mean_shift(ref: np.ndarray, new: np.ndarray) -> float:14    """The distribution layer: how far the centre moved, as a ratio."""15    return float(ref.mean() / new.mean())16
17print(schema_ok(batch), range_ok(batch))         # -> True True   both cheap checks pass18print(int(reference.mean()), int(batch.mean()))  # -> 4850 4819print(round(mean_shift(reference, batch), 1))    # -> 99.8        the layer that catches it

Three assertions, one of which earns its keep. It's also the one that costs you a reference window and a threshold, which is why it's the one that doesn't get written.

Watch Out For

Validating the schema and stopping there

pandera, great_expectations or a Pydantic model, checking columns and types, running green for months.

Type checks catch a class of failure that mostly announces itself anyway — a string in a numeric column breaks something downstream within minutes. What they cannot see is any change that preserves the type: cents to pounds, kilograms to pounds, a categorical value renamed from active to ACTIVE, a timezone switched from UTC to local, an ID space reset so yesterday's 1001 and today's 1001 are different customers.

Every one of those is a valid value of the right type, so a schema suite reports a clean batch and every number downstream is wrong.

Add three per numeric column — mean, a quantile, and the null rate — and two per categorical column — the distinct count and the top few frequencies — each against a stored reference. It's a dozen lines per table and it's the difference between a suite that catches things and one that decorates the repository.

Checks strict enough that someone turns them off

A fail-closed assertion that the daily mean stays within 20% of the reference. It's correct in April. In late November the traffic mix shifts, the check fires, the pipeline halts, and it's the on-call engineer's problem at 2am for four nights running.

On night five it gets commented out, with # TODO: revisit above it. Nobody revisits, and now that column has no monitoring at all. You are worse off than if you'd never written it, because the absence looks like coverage.

Design for the incentive. Distribution checks warn and route to a queue rather than blocking. Blocking thresholds sit where no legitimate event reaches them — a hundredfold mean shift, a null rate going from 2% to 90%, a row count at 3% of normal. Give every check a severity and an owner, and treat a check being disabled as an incident in itself.

Then review the noisy ones on a schedule. A check that fires monthly and is always benign is miscalibrated, and fixing the threshold is the work.

The Quick Version

A change that breaks the pipeline is the good case. The expensive one keeps it running and changes what the numbers mean.
Four layers: schema, values, distributions, cross-field and cross-batch.
The distribution layer catches unit changes and renamed categories, and it's the one that gets skipped.
Gate at pipeline boundaries and at both the training and serving entry points.
Schema and value checks block. Distribution checks warn, and only block at thresholds no real event reaches.
A validation suite you own is a tripwire. A contract in the producer's repository is prevention.
Cents to pounds passed schema, range and cross-field checks. Only the mean ratio of 99.8 noticed.
A check that gets disabled at 2am leaves you worse off than one you never wrote, because the gap looks like coverage.