Skip to content
AI360Xpert
Data Concepts

Dataset Documentation

Six months on, nobody remembers where the rows came from or what the label meant. A data card is the artefact that answers that without the person who built it.

A data card answers six questions about a dataset, and the label definition is the one that silently changes what every model trained on it was actually predicting
A data card answers six questions about a dataset, and the label definition is the one that silently changes what every model trained on it was actually predicting

Why Does This Exist?

A model has been in production for a year and it predicts churn. Someone asks what churn means.

Three answers come back. The data engineer says no purchase in 90 days, because that's the SQL. A product manager says cancelled subscription. An analyst says no login in 30 days, because that's the dashboard definition. All three are in use, in different places, and the training label came from one of them.

Nobody was careless. The person who wrote the label knew exactly what they meant and left within eighteen months, which is the normal tenure of anyone in this position. The knowledge was never in the dataset, only next to it.

That's what documentation is for here — not compliance, not tidiness. It's the mechanism by which a dataset stays usable after the person who built it is gone, and it's the difference between an asset and a liability with rows in it.

Think of It Like This

A museum specimen with no label

A natural history collection holds two beetles in adjacent drawers.

The first has a card: species, where it was collected, the date, who collected it, the method, and a note that the left antenna was damaged in transit. A researcher a century later can use it in a study, because they know what it is and what's wrong with it.

The second is a beetle in a box. Nobody knows where it came from, when, or whether it's even from this continent. It's a beautiful specimen and it's scientifically worthless — not because anything is wrong with it, but because nothing about it can be verified.

The beetles are identical. One is data and one is an object, and the difference is a card.

How It Actually Works

A data card is a short document that lives with the dataset, in version control, updated when the dataset changes. Six things it has to answer.

1. Provenance

Where the rows came from: which systems, which time range, which query or scrape or vendor. Enough that someone could rebuild it, and enough that when a source turns out to be broken you know which datasets are affected. Pair it with a version identifier so "which rows" and "where from" are both answerable — see data versioning.

2. Collection method

How the rows arrived, because that determines what the data can support. Was this an experiment or an observation? Self-reported or measured? A convenience sample of whoever happened to be online, or something closer to representative? This section is where selection bias gets written down, and it's usually the only place it exists.

3. Label definition

The one that causes the most damage, so give it the most room. Not "the churn flag" — the exact rule, the cutoff, the edge cases, who decided, and when it last changed. If humans labelled it, the guidelines and the agreement rate belong here, which is what labelling and annotation produces.

A label whose definition changed mid-collection is two labels sharing a column name, and no amount of modelling recovers from that.

4. Coverage and gaps

What's in it and what isn't. Which regions, languages, devices, date ranges, customer segments. Which groups are thin, and what the class balance is. A model is only reliable where its training data lived, so the gaps are the operating limits — this is the honest version of the manifold argument written for a stakeholder.

What you're allowed to do with it. Licence terms, whether consent covers model training, whether it can leave a jurisdiction, whether it can be used commercially, and the retention deadline. This is the section that stops a project at review, and it's much cheaper to answer before the model is built. Data privacy and governance is the depth.

6. Known limitations

Everything you already know is wrong: the sensor that drifted in March, the region where the join is unreliable, the duplicate cluster, the period where a bug halved the event volume. Writing these down feels like admitting failure and it's the highest-value section in the document, because it's the only one nobody can reconstruct later.

Keeping it honest

Two rules make the difference between a card and a formality. Version it with the data, in the same repository, reviewed in the same pull request — a wiki page drifts within a quarter. And write the limitations before the results, because they get quietly dropped once there's a number to protect.

Worked example

Eight customers. Three churn definitions, applied to the same rows.

"No purchase in 90 days" flags 5. "Cancelled subscription" flags 2. "No login in 30 days" flags 4. The purchase and login definitions agree on 7 of 8 customers, which is close enough to look interchangeable on a spot check and different enough to move a model.

Scale that to 50,000 customers and the definitions differ on thousands of rows. Two teams reporting "churn accuracy" are then reporting on two different targets, their numbers aren't comparable, and nothing in either pipeline says so.

One paragraph in a data card prevents all of it.

Show Me the Code

import numpy as np
days_since_purchase: np.ndarray = np.array([10, 100, 200, 45, 120, 95, 5, 400])cancelled: np.ndarray = np.array([0, 0, 1, 0, 0, 0, 0, 1])days_since_login: np.ndarray = np.array([2, 40, 210, 8, 35, 12, 1, 380])
def by_purchase(d: np.ndarray, cutoff: int = 90) -> np.ndarray:    return d > cutoff
def by_login(d: np.ndarray, cutoff: int = 30) -> np.ndarray:    return d > cutoff
print(int(by_purchase(days_since_purchase).sum()))   # -> 5  "no purchase in 90 days"print(int(cancelled.sum()))                          # -> 2  "cancelled the subscription"print(int(by_login(days_since_login).sum()))         # -> 4  "no login in 30 days"print(int((by_purchase(days_since_purchase) == by_login(days_since_login)).sum()))  # -> 7

Three functions, one word, three targets. The last line is why a spot check doesn't catch it.

Watch Out For

A label definition that changed mid-collection

The churn cutoff moved from 60 days to 90 in an April refactor. The column name didn't change, so the historical rows keep the old rule and everything after April uses the new one.

Now the column holds two labels. A model trained across the boundary learns an average of two different targets, and its errors cluster in a way that looks like concept drift — because functionally it is, except the world didn't change, your definition did.

The reason it survives is that nothing is inconsistent. Every row is correctly labelled under the rule in force when it was written. There's no null, no contradiction, no failing check.

Two defences. Record the label rule with a valid-from date in the card, so a definition change is a documented event rather than a commit message. And when you find one after the fact, either relabel history under the current rule or cut the training window at the boundary. Averaging across it is the one option that can't be defended.

Documenting the dataset and not its gaps

The card lists provenance, schema, collection method and licence. It's genuinely useful. It says nothing about what isn't in the data.

So the model ships, and it turns out the training set is 94% one country, has almost no rows from the newest device type, and covers a period that excludes the last pricing change. None of that is a defect — it's what the data is — and every one of them is an operating limit that nobody stated, so nobody checked.

The pattern that follows is predictable: the model performs as advertised in aggregate and badly on a segment somebody cares about, and the review asks why nobody knew.

Write the coverage section as a table of counts by the dimensions that matter — region, language, segment, device, date — and mark anything under a threshold as thin. Then write the limitations section before you have results, and require it to be non-empty. A dataset with no known limitations means nobody looked.

The Quick Version

  • The knowledge that makes a dataset usable is rarely in the dataset. Documentation is how it survives the person who built it.
  • Six sections: provenance, collection method, label definition, coverage and gaps, licence and consent, known limitations.
  • The label definition does the most damage. Write the exact rule, its edge cases, who set it, and when it last changed.
  • One word, three definitions: on eight customers, "churn" flagged 5, 2 and 4, with two definitions agreeing on 7 of 8.
  • A label whose rule changed mid-collection is two labels in one column, and averaging across the boundary is indefensible.
  • Coverage gaps are operating limits. A model is only reliable where its data lived.
  • Licence and consent stop projects at review. Answer them before the model exists.
  • Known limitations is the highest-value section, because it's the only one nobody can reconstruct later.
  • Version the card with the data and write limitations before results, or both drift.

Related concepts