Data Quality Monitoring

Why Does This Exist?

Old software fails loudly. A missing field throws, a type mismatch crashes, and a stack trace points at the line. Machine learning fails quietly. If a pipe replaces income with nulls or shifts ages by a column, the model still multiplies and returns a confident wrong score.

The model assumes production rows look like training rows in shape and in stats. Schema renames, failed joins, stuck sensors, and slow behavior drift all break that promise. Without checks at the entry, the first sign is a business metric sliding weeks later. This page covers entry checks that halt bad batches. What changed after launch lives in model drift detection, the live score and latency signals live in model monitoring, and the wasted GPU behind replays is why cost optimization matters.

Basics first: a schema is the promised columns and types. A null is a missing value a model cannot score without a fill. Drift is a slow stat shift from the training mix. If those three are new, read this page slowly because everything below stacks them.

Think of It Like This

How It Actually Works

Checks run on every batch before inference, in three layers from cheap to statistical. Thresholds come from training, not from guesses.

Schema and shape checks

Confirm columns, order, and types. A renamed user_age to age_years, an income column arriving as text, or a dropped status column must fail fast. These checks are exact and cost almost nothing. They catch deploys, not drift.

Completeness checks

Measure the share of nulls per critical field. A failed join or a timed-out fetch shows up as a null spike, not as an error. Cardinality also matters: a status field with 3 allowed values that suddenly shows 400 distinct strings is corrupt even when nulls look fine.

Range and distribution checks

Bound numbers to possible values and compare stats to training. Ages must sit inside a plausible band. Amounts must stay non-negative. A slow mean shift or a new hump in the histogram signals drift before accuracy moves.

Worked example: 5 rows, 3 failures

Five production rows arrive with fields age, income, and status. Training allowed status in a set of 3 values, nulls below 5 percent, and ages from 18 to 99.

Rows: 25 with 50000 active, 34 with 60000 active, 45 with null closed, 120 with 75000 active, minus 5 with 45000 unknown.

Null share:

Limit is 5 percent, so the batch fails completeness by 15 points. Age check fails twice: 120 above 99 and minus 5 below 18. Status check fails once: unknown is outside the set of 3. Two rows are clean, three are held, and the batch halts even though no service threw an error. That is the diagram above: 20 percent nulls against a 5 percent limit and two ages outside 18 to 99.

Alerting by severity

A small drift on a minor field logs to a dashboard and scoring continues. A null column or a schema break halts the pipe, pages the owner, and falls back to a safe default or cached score until the pipe is fixed. Every halt stores the batch and the failed check for replay.

Code

The point is the gate: pure stdlib checks for null share, range, and allowed set, with the same 5 rows as the worked example.

1rows: list[dict] = [2    {"age": 25, "income": 50000, "status": "active"},3    {"age": 34, "income": 60000, "status": "active"},4    {"age": 45, "income": None, "status": "closed"},5    {"age": 120, "income": 75000, "status": "active"},6    {"age": -5, "income": 45000, "status": "unknown"},7]8
9allowed = {"active", "closed", "suspended"}10
11
12def null_share(field: str) -> float:13    return sum(1 for r in rows if r[field] is None) / len(rows)14
15
16def bad_ages(lo: int, hi: int) -> list[int]:17    return [int(r["age"]) for r in rows if not (lo <= int(r["age"]) <= hi)]18
19
20def bad_status() -> list[str]:21    return [str(r["status"]) for r in rows if str(r["status"]) not in allowed]22
23
24print(f"{null_share('income'):.2f}")  # -> 0.2025print(bad_ages(18, 99))  # -> [120, -5]26print(bad_status())  # -> ['unknown']

One null in five prints 0.20 against a 0.05 limit. Two ages print outside 18 to 99. One status prints outside the set of 3. Change the limit to 0.25 and the null check passes, which shows why limits must come from training impact rather than round numbers.

Watch Out For

The Quick Version

• Models fail quietly on bad rows, so checks must run on every batch before inference.
• Schema checks catch renames and type flips from deploys at almost no cost.
• One null in five is 20 percent against a 5 percent limit, so the batch halts on completeness alone.
• Ages minus 5 and 120 fail an 18 to 99 band, and unknown fails a set of 3 allowed statuses.
• Two clean rows score while three held rows page, log, and fall back instead of poisoning predictions.
• Limits come from training impact and rolling windows, not from round numbers that growth will cross.

What to Read Next

• Model Drift Detection
• Cost Optimization in ML
• Model Monitoring Platform: Drift to Alert