Skip to content
AI360XpertBeta

Dataset Documentation

Six months on, nobody remembers where the rows came from or what the label meant. A data card is the artefact that answers that without the person who built it.

Three definitions of churn flag 5, 2 and 4 of the same 8 customers, so the label rule written in the data card decides what every model trained on the column predicts.
Three definitions of churn flag 5, 2 and 4 of the same 8 customers, so the label rule written in the data card decides what every model trained on the column predicts.

Why Does This Exist?

A model has been in production for a year and it predicts churn. Someone asks what churn means.

Three answers come back. The data engineer says no purchase in 90 days, because that is the SQL. A product manager says cancelled subscription. An analyst says no login in 30 days, because that is the dashboard definition. All three are in use, in different places, and the training label came from one of them.

Nobody was careless. The person who wrote the label knew exactly what they meant and left within eighteen months, which is the normal tenure of anyone in this position. The knowledge was never in the dataset, only next to it. That is what documentation is for here. Not compliance, not tidiness. It is the mechanism by which a dataset stays usable after the person who built it is gone, and it is the difference between an asset and a liability with rows in it.

The card lives with the data, so it pairs with data versioning. The label section draws on whatever labelling and annotation produced. The licence section is answered before the model exists, with data privacy and governance as the depth. This page covers the six sections a card must answer.

Think of It Like This

A museum specimen with no label

A natural history collection holds two beetles in adjacent drawers.

The first has a card: species, where it was collected, the date, who collected it, the method, and a note that the left antenna was damaged in transit. A researcher a century later can use it in a study, because they know what it is and what is wrong with it.

The second is a beetle in a box. Nobody knows where it came from, when, or whether it is even from this continent. It is a beautiful specimen and it is scientifically worthless. Not because anything is wrong with it, but because nothing about it can be verified.

The beetles are identical. One is data and one is an object, and the difference is a card.

Where it stops: a museum card describes one object that never changes. A data card describes a pipeline that drifts, gets relabelled, and gets joined to new sources, so the card needs a valid-from date and a version, not just a description.

How It Actually Works

A data card is a short document that lives with the dataset, in version control, updated when the dataset changes. Six things it has to answer.

1. Provenance

Where the rows came from: which systems, which time range, which query or scrape or vendor. Enough that someone could rebuild it, and enough that when a source turns out to be broken you know which datasets are affected. Pair it with a version identifier so "which rows" and "where from" are both answerable.

2. Collection method

How the rows arrived, because that determines what the data can support. Was this an experiment or an observation. Self-reported or measured. A convenience sample of whoever happened to be online, or something closer to representative. This section is where selection bias gets written down, and it is usually the only place it exists.

3. Label definition

The one that causes the most damage, so give it the most room. Not "the churn flag". The exact rule, the cutoff, the edge cases, who decided, and when it last changed. If humans labelled it, the guidelines and the agreement rate belong here. A label whose definition changed mid-collection is two labels sharing a column name, and no amount of modelling recovers from that.

4. Coverage and gaps

What is in it and what is not. Which regions, languages, devices, date ranges, customer segments. Which groups are thin, and what the class balance is. A model is only reliable where its training data lived, so the gaps are the operating limits.

What you are allowed to do with it. Licence terms, whether consent covers model training, whether it can leave a jurisdiction, whether it can be used commercially, and the retention deadline. This is the section that stops a project at review, and it is much cheaper to answer before the model is built.

6. Known limitations

Everything you already know is wrong: the sensor that drifted in March, the region where the join is unreliable, the duplicate cluster, the period where a bug halved the event volume. Writing these down feels like admitting failure and it is the highest-value section in the document, because it is the only one nobody can reconstruct later.

Keeping it honest

Two rules separate a card from a formality. Version it with the data, in the same repository, reviewed in the same pull request. A wiki page drifts within a quarter. And write the limitations before the results, because they get quietly dropped once there is a number to protect.

Worked example

Eight customers. Three churn definitions, applied to the same rows.

"No purchase in 90 days" flags 5. "Cancelled subscription" flags 2. "No login in 30 days" flags 4. The purchase and login definitions agree on 7 of 8 customers, which is close enough to look interchangeable on a spot check and different enough to move a model.

Scale that to 50,000 customers and the definitions differ on thousands of rows. Two teams reporting "churn accuracy" are then reporting on two different targets, their numbers are not comparable, and nothing in either pipeline says so. One paragraph in a data card prevents all of it.

Code

import numpy as np
days_since_purchase: np.ndarray = np.array([10, 100, 200, 45, 120, 95, 5, 400])cancelled: np.ndarray = np.array([0, 0, 1, 0, 0, 0, 0, 1])days_since_login: np.ndarray = np.array([2, 40, 210, 8, 35, 12, 1, 380])
def by_purchase(d: np.ndarray, cutoff: int = 90) -> np.ndarray:    return d > cutoff
def by_login(d: np.ndarray, cutoff: int = 30) -> np.ndarray:    return d > cutoff
print(int(by_purchase(days_since_purchase).sum()))  # -> 5print(int(cancelled.sum()))                         # -> 2print(int(by_login(days_since_login).sum()))        # -> 4print(int((by_purchase(days_since_purchase) == by_login(days_since_login)).sum()))  # -> 7

Three functions, one word, three targets. The last line is why a spot check does not catch it.

Watch Out For

A label definition that changed mid-collection

The churn cutoff moved from 60 days to 90 in an April refactor. The column name did not change, so the historical rows keep the old rule and everything after April uses the new one.

Now the column holds two labels. A model trained across the boundary learns an average of two different targets, and its errors cluster in a way that looks like concept drift. Functionally it is drift, except the world did not change, your definition did.

Symptom: nothing is inconsistent. Every row is correctly labelled under the rule in force when it was written, so there is no null, no contradiction, no failing check. Fix: record the label rule with a valid-from date in the card, so a definition change is a documented event rather than a commit message. Found after the fact, either relabel history under the current rule or cut the training window at the boundary. Averaging across it is the one option that cannot be defended.

Documenting the dataset and not its gaps

The card lists provenance, schema, collection method and licence. Genuinely useful. It says nothing about what is not in the data.

So the model ships, and the training set turns out to be 94 percent one country, with almost no rows from the newest device type, covering a period that excludes the last pricing change. None of that is a defect. It is what the data is. Every one of them is an operating limit that nobody stated, so nobody checked.

Symptom: the model performs as advertised in aggregate and badly on a segment somebody cares about, and the review asks why nobody knew. Fix: write the coverage section as counts by the dimensions that matter (region, language, segment, device, date) and mark anything under a threshold as thin. Then write the limitations section before you have results, and require it to be non-empty. A dataset with no known limitations means nobody looked.

The Quick Version

  • The knowledge that makes a dataset usable is rarely in the dataset. Documentation is how it survives the person who built it.
  • Six sections: provenance, collection method, label definition, coverage and gaps, licence and consent, known limitations.
  • The label definition does the most damage. Write the exact rule, its edge cases, who set it, and when it last changed.
  • One word, three definitions: on eight customers, "churn" flagged 5, 2 and 4, with two definitions agreeing on 7 of 8.
  • A label whose rule changed mid-collection is two labels in one column, and averaging across the boundary is indefensible.
  • Coverage gaps are operating limits. A model is only reliable where its data lived.
  • Licence and consent stop projects at review. Answer them before the model exists.
  • Known limitations is the highest-value section, because it is the only one nobody can reconstruct later.
  • Version the card with the data and write limitations before results, or both drift.

What to Read Next