Pipeline Orchestration: DAGs That Resume

Orchestration turns scattered ML scripts into a dependency graph of containerized steps, so when the 38 minute train step fails only it reruns while the finished 4 minute prep result is reused.

A three step graph runs prep once in 4 minutes, retries the failed 38 minute train step alone, and finishes evaluation in 3 minutes for 80 minutes instead of 84.

Why Does This Exist?

Early ML work runs as a stack of scripts: one pulls data, one cleans it, one trains, one evaluates. That works until the 38 minute training run dies at minute 35 and the only option is rerunning everything from the top, or until nobody can say which data built which model.

Orchestration replaces the pile with a graph the machine understands: steps as containerized tasks, edges as dependencies, artifacts passed by reference. The scheduler from containerization gives each step its sealed box, runs get recorded through experiment tracking, and the whole graph ships inside CI/CD pipelines. The payoff is resume: a failure restarts the failed step, not the week.

The one idea you need first

A directed acyclic graph, a DAG, is tasks joined by arrows with no loops. The arrows say what must finish first. No loops means the scheduler can always find something runnable, and can prove the full order before anything starts.

Think of It Like This

A dinner rush run by the ticket rail

Orders hang on a rail in the order they arrived. The grill cook starts the steak, the salad cook plates greens, and the expeditor holds the table until every dish is ready. If a sauce breaks, only the sauce pan restarts. Nobody re-fires the steak that already rested perfectly.

The mapping is direct. Tickets are the DAG. Stations are containerized tasks. The rail order is the dependency list. The expeditor is the orchestrator, tracking pending, running, and done. Re-firing one sauce is the retry of one failed step with cached upstream results.

Where it stops: a kitchen ticket lives for minutes and a ruined dish costs little. A pipeline step burns GPU hours and its outputs must be stored, versioned, and auditable, which is why artifact passing matters here and a shout across the pass does not cut it.

How It Actually Works

You declare each step with its inputs, its container image, and its outputs. The orchestrator sorts the graph topologically, runs whatever has no unfinished parents, and records every state change. Downstream steps receive artifact references, never raw blobs, so lineage survives contact with reality. The trained model artifact lands in the model registry, where one atomic promotion moves it to Production and archives the incumbent.

The run that failed politely

Daily training has three steps. Prep cleans and splits data in 4 minutes. Train fits the model in 38 minutes. Evaluate scores it in 3 minutes. Evaluate needs both the model and the clean data, so it waits on two parents.

First attempt: prep succeeds in 4, train dies at minute 35 on a spot eviction. Wasted so far: $4 + 35 = 39$ minutes. The scheduler marks downstream as blocked, keeps the prep artifact, and retries train alone: 38 more minutes, then evaluate runs 3. Total wall time is $4 + 35 + 38 + 3 = 80$ minutes. Without caching, prep would rerun and the total would be $4 + 35 + 4 + 38 + 3 = 84$ . The saved 4 minutes look small until the prep step grows into a 2 hour feature build, at which point resume is the whole ballgame.

Why the graph must stay acyclic

A cycle means step A waits on B while B waits on A, and nothing can ever start. The scheduler detects this at parse time with a topological sort and refuses the pipeline before it burns a cent. The fix is always structural: break the loop by moving the shared work into its own upstream step both sides depend on.

Code

The point of this snippet is the ordering plus the resume: a topological sort decides the sequence, and a retry reruns only the failure.

1from collections import deque2
3
4def topo_order(tasks: dict[str, list[str]]) -> list[str]:5    """Kahn's algorithm over task-to-parents mapping."""6    children: dict[str, list[str]] = {task: [] for task in tasks}7    pending: dict[str, int] = {}8    for task, parents in tasks.items():9        pending[task] = len(parents)10        for parent in parents:11            children[parent].append(task)12    ready = deque(sorted(t for t, n in pending.items() if n == 0))13    order: list[str] = []14    while ready:15        task = ready.popleft()16        order.append(task)17        for child in sorted(children[task]):18            pending[child] -= 119            if pending[child] == 0:20                ready.append(child)21    if len(order) != len(tasks):22        raise ValueError("cycle detected: no valid order exists")23    return order24
25
26graph = {"prep": [], "train": ["prep"], "evaluate": ["train", "prep"]}27print(topo_order(graph))  # -> ['prep', 'train', 'evaluate']28
29durations = {"prep": 4, "train": 38, "evaluate": 3}30wasted = durations["prep"] + 35  # first train attempt dies at minute 3531total = wasted + durations["train"] + durations["evaluate"]32print(total)  # -> 80

Order first, minutes second: 80 with caching against 84 without, and the gap grows with the prep step.

Watch Out For

One giant task wearing a DAG costume

A thousand-line script doing extraction, cleaning, and training gets wrapped as a single node. When training fails at minute 35, the retry reruns extraction and cleaning too, because the scheduler cannot see inside the box.

The symptom is retries that cost as much as full reruns. Fix it by splitting along failure and cache boundaries: one step per artifact, each with a single responsibility, so the retry boundary matches the work boundary.

A dependency loop that never starts

Step A reads B's output while B reads A's, often through a shared config nobody noticed. The symptom is a pipeline that parses, schedules nothing, and reports a topological sort error.

Fix it by extracting the shared piece into its own upstream step both depend on. Then add a parse-time check to continuous integration so the loop fails the pull request instead of the nightly run.

The Quick Version

Orchestration declares ML work as a DAG of containerized tasks with explicit dependencies and artifact passing.
The scheduler runs steps whose parents succeeded, blocks the rest, and retries only failures against cached upstream results.
A failed 38 minute train step costs a retry of 38 minutes, not a rerun of the whole pipeline.
The graph must stay acyclic, or no step can ever start, so cycles fail fast at parse time.
Split steps along cache boundaries, or retries will cost as much as the reruns they replace.