ML Pipeline Architecture: From Data to Registry
A notebook proves an idea once. A pipeline replays ingestion, preparation, training, evaluation and registration on schedule so the same code produces a fresh model without manual steps.
Why Does This Exist?
A churn notebook scores 0.87 on a laptop and everyone celebrates. Next month the data changes, the author is on leave, and nobody can rerun the cells in order because one cell depended on a variable set three runs ago. Retraining becomes a rescue project instead of a schedule.
The failure is not the model. It is the manual path from raw rows to a deployable artifact. ML workflow names the loop from framing to monitoring. This page covers the machinery that makes the loop repeatable: a directed acyclic graph of isolated steps where data flows one way, each step reruns from its inputs, and a gate stops a worse model before it ships.
Think of It Like This
A bakery line with five stations
A commercial bakery runs five stations in order. Station one weighs flour. Station two mixes for four minutes. Station three bakes at one fixed heat. Station four tastes one slice against a written standard and stops the line when it fails. Station five boxes the loaf with its recipe card and batch number.
Weighing maps to ingestion, mixing maps to preparation, baking maps to training, tasting maps to evaluation, and boxing with the card maps to registration. The recipe card is versioned code plus parameters. The batch number is the model version.
Where it stops: loaves do not drift after boxing and stations do not learn from their own output. Models degrade in the wild and their predictions change the next batch of data, so a pipeline needs the retraining schedule and the monitoring watch that a bakery never does.
How It Actually Works
Call a run one pass from raw rows to a registered artifact. Call a node one isolated step with declared inputs and outputs. Call an edge the data dependency that forces order. The graph is acyclic, meaning no step waits on its own output, so a scheduler can retry any node without rerunning everything upstream.
The five stages in order
Ingestion pulls a fixed window on schedule. Preparation cleans and encodes with saved parameters. Training fits weights on the prepared output. Evaluation scores the fresh weights on a held out set and compares against the production champion. Registration stores weights, preparation parameters, code version and metrics as one versioned artifact. Feature stores keep the preparation numbers identical online and offline. Model registry is the shelf where that artifact waits. Experiment tracking is the log that says which settings produced it.
Worked example: one weekly run with real counts
The Sunday job pulls the last 30 days of events:
Preparation drops rows with no label and fills the rest:
Training fits once on 12,220 rows. Evaluation scores 2,000 held out rows. The new weights get 1,740 right. The production champion gets 1,700 right:
The gain is:
The gate needs better than 0.85, so 0.87 passes by 0.02 and the run registers version 7 as weights plus scaler means plus encoder version. Had the new score read 0.84, the graph would halt and version 6 would stay live. That halt is the diagram headline: five boxes left to right, counts falling from 12,400 to 12,220, then 0.87 against 0.85 at the gate.
Why isolation matters
Each node reads only its declared inputs. The scaler fitted in preparation is saved with its mean and variance and reused at serving time. Nothing is refit from memory. When a run fails, the scheduler retries that node alone because no hidden notebook state leaks between steps.
Code
The point is the gate: rebuild the holdout check from the worked example and confirm the promote decision.
def gate_decision(new_correct: int, old_correct: int, total: int, bar: float = 0.85) -> str: """Decide promotion from holdout counts. Returns promote or hold.""" new_score: float = new_correct / total old_score: float = old_correct / total if new_score > bar and new_score > old_score: return f"promote {new_score:.2f} over {old_score:.2f}" return f"hold {new_score:.2f} vs {old_score:.2f}"
print(gate_decision(1740, 1700, 2000)) # -> promote 0.87 over 0.85print(gate_decision(1680, 1700, 2000)) # -> hold 0.84 vs 0.85The printout matches the hand computation: 0.87 clears both the 0.85 bar and the 0.85 champion, while 0.84 fails and keeps version 6 live. Change the bar to 0.90 and watch the 0.87 run switch from promote to hold.
Watch Out For
Refitting the scaler at serving time
Preparation saves a scaler mean of 48.2 and variance from training, but the serving path recomputes them on live traffic. Offline holdout reads 0.87 while live accuracy sags to 0.71 because the numbers entering the model are shifted.
Symptom: a gap between holdout and live with no code change, widest on scaled numeric features. Fix: save preparation parameters as part of the registered artifact and load them at serving. Use feature stores when online and offline paths must read identical values.
A gate that compares against a stale champion
The champion has sat live for four months while tastes shifted. The challenger scores 0.86 against a champion at 0.83, passes, and ships, but both are measured on last quarter holdout rows that no longer match traffic.
Symptom: the gate passes every week while business metrics slide. Fix: refresh the holdout window on schedule, require the challenger to beat the champion on recent rows, and let model monitoring force a reframe when the input mix moves.
The Quick Version
- A pipeline is a directed acyclic graph of isolated steps that reruns ingestion, preparation, training, evaluation and registration on schedule.
- Our weekly run pulls 12,400 rows, drops 180 nulls, and keeps 12,220 for training.
- The fresh model scores 1,740 of 2,000 for 0.87 against the champion at 1,700 of 2,000 for 0.85, so it clears the 0.85 gate by 0.02.
- Registration stores weights plus preparation parameters plus code version as one artifact, which is what serving loads.
- Isolation plus a fresh holdout is what keeps a notebook result repeatable in production.