ML Workflow: Six Stages in a Loop
Problem, data, model, evaluation, shipping and monitoring run in a loop, not a line. What monitoring finds reframes the problem instead of only retraining the model.
Why Does This Exist?
A churn model hits 89 percent in a notebook and ships. Three months later marketing complains it flags loyal customers while real churners slip through. Nobody touched the code. A new pricing tier launched, a competitor arrived, and the meaning of likely to churn shifted under a model frozen at the old meaning.
The notebook number was honest and measured the wrong moment: fit on data from a past that had already ended. What is machine learning explains fitting a function to examples. This page covers what must happen before and after that fit so the function stays worth trusting a month later, and how model evaluation differs from the watch that follows shipping.
Think of It Like This
A restaurant menu that never freezes
A restaurant opens with a guess about what diners want. It watches which dishes sell, drops what nobody orders, tweaks portions that return half eaten, and sometimes learns the whole concept is off, not one recipe.
Guess maps to problem framing, sourcing maps to data collection, cooking maps to training, tasting maps to evaluation, opening night maps to shipping, and nightly sales maps to monitoring. The loop never closes while doors stay open, and a bad week can mean rethinking the concept rather than adding salt.
Where it stops: diners tell you plainly when a dish fails. Model inputs drift silently and predictions reshape the next crowd, so monitoring must measure distributions before complaints arrive, not just count empty plates.
How It Actually Works
Call framing the turn from a business question into a target, a metric and a decision about what the prediction will trigger. Call monitoring the continuous watch on live inputs and behavior after shipping. Training is the smallest calendar item on a mature team because its recipe is the clearest, while data quality and drift have no fixed endpoint.
The six stages briefly
Framing sets target, metric and action. Data collection gathers examples plus a plan for future labels. Training fits the function. Evaluation scores held out data once before launch. Shipping puts the model where it changes a decision. Monitoring watches live behavior because nothing before launch shows what happens once the world reacts to the predictions. Experiment tracking keeps training runs comparable. The machinery that replays the middle stages lives in ml pipeline architecture.
Why the picture is a loop
Monitoring often reports that framing itself aged, not that weights need a refresh. The churn case needed a new definition of churn after pricing changed, not more rows under the old definition. The arrow back to stage one is the whole point: retraining on fresh data under a stale target preserves the error faster.
Worked example: a pricing shift caught by inputs alone
Track average order value. At training time the mean is 50 with standard deviation 5 over 1,000 orders. Months later two windows arrive, each 200 orders. The stable window averages 51. The post pricing window averages 62. Flag when the live mean moves more than 0.5 training deviations:
No prediction is inspected and no fresh label is needed. The 2.4 sigma move trips the watch before accuracy can even be measured, which is the early warning role evaluation cannot play because evaluation runs once on the past.
Code
The point is the early warning: flag a live window from input shape alone, with no labels and no model call.
def shift_flag(train_mean: float, train_std: float, live_mean: float, limit: float = 0.5) -> str: """Flag a live window when its mean moves more than limit training deviations.""" shift: float = abs(live_mean - train_mean) / train_std return f"shift {shift:.1f} flagged {str(shift > limit)}"
print(shift_flag(50.0, 5.0, 51.0)) # -> shift 0.2 flagged Falseprint(shift_flag(50.0, 5.0, 62.0)) # -> shift 2.4 flagged TrueThe printout matches the hand division: 0.2 stays quiet under the 0.5 limit while 2.4 trips it. Change the limit to 3.0 and watch the pricing shift go quiet, which shows why the threshold is a product choice rather than math.
Watch Out For
Treating shipping as the finish line
The team staffs framing through launch and reassigns everyone after. The world moves, inputs drift, and the first alarm is a quarterly business review with no logs to explain it.
Symptom: steady code with sliding decisions and no owner for the live watch. Fix: staff monitoring from day one with named thresholds and a retraining owner, as model monitoring describes, rather than adding dashboards after visible harm.
Retraining on fresh data under a stale target
Monitoring flags the churn slide and the reflex is a fresh fit on last month. Accuracy on the old target rises while the business meaning keeps sliding, because likely to churn now includes annual plan hoppers the old label never counted.
Symptom: retrains that score well and change nothing in the product. Fix: recheck framing before refitting, rewrite the target when its meaning moved, and use the bias variance tradeoff only after the target is honest again.
The Quick Version
- The ML workflow runs framing, data, training, evaluation, shipping and monitoring as a loop, not a line.
- Monitoring watches live input shape continuously, while evaluation scores held out data once before shipping.
- Our stable window shifts 0.2 training deviations and stays quiet, while the pricing window shifts 2.4 and trips the 0.5 limit.
- What monitoring surfaces often belongs back in framing, not in a blind retrain.
- Data work and live watch take more calendar time than fitting, because only fitting has a fixed recipe.