I read MLOps maturity models from the top down

The level 0 → level 3 staircase describes the destination well, and the order of the steps is made up. Following it to the letter is the best way to automate a model nobody uses. My own order, and the small calculation behind it.

Every vendor has its maturity model. Level 0: everything by hand. Level 1: automated training. Level 2: automated deployment. Level 3: continuous, monitored, self-healing. A neat staircase, usually with the vendor's product on the top step.

I agree with the destination. What bothers me is the order of the steps, and the order is the one thing a team can actually act on.

What these models get right

The capabilities they list are real. Reproducible training, automated deployment, monitoring, rollback: a team that has all of them ships with more confidence than a team that has none. There is no filler in the list.

They also give a shared vocabulary. Saying "we are at level 1" in a budget meeting lands better than a paragraph of nuance, especially with a non-technical audience.

And they make invisible work visible. A good part of MLOps cannot be demoed. A maturity model is one of the few tools that gets "we cannot roll back" across to someone who does not feel the risk.

What bothers me

Levels are presented as prerequisites. They are mostly independent capabilities, stacked on top of each other by the drawing. Monitoring requires no automated retraining. Rollback works perfectly well without a feature store. Drawn as steps, they lead teams to build level 2 automation before checking that the model deserves to be deployed at all.

They measure automation. Levels count how much of the pipeline runs without a human. That is an indirect, sometimes decent, signal of what really matters: how long it takes to see a problem, and what it costs to undo it.

Maturity levels drawn as a staircase, compared with the same capabilities plotted by cost to acquire against risk reducedthe drawingmanualauto-trainauto-deploycontinuousimplies: in this orderwhat actually guides the ordercost to acquire →risk reduced ↑rollbackserved-input loggingoutput monitoringreproducible trainingautomated retraining
Figure 1. The same capabilities, placed by cost and by the risk they remove. Top left: cheap and decisive. Automated retraining sits bottom right, and it is often the one maturity models put in the spotlight.

They forget what the model serves. A level 3 platform serving a model nobody uses is worth less than a level 0 notebook that people depend on. Maturity models score the machinery, never what it carries.

They suit the vendor. The top of the staircase is almost always shaped like a product. That does not make it wrong, but it explains why "buy a platform" shows up so early in the sequence.

The three questions I ask instead

How long before we know something broke? That is time to detection. If the answer is "when a user complains", the rest of the list can wait.

How long before it is fixed? That is time to recovery. In practice: can we roll back, and have we actually done it.

How long before we understand why? That is time to diagnosis. This is where logging what was actually served pays off, incident after incident.

Any on-call engineer recognises these questions. Machine learning brings nothing new on that side, it just adds a few ways to fail without raising an exception.

The maths, intuitively: damage is an area

The three questions come down to a simple calculation. During an incident, service quality drops by some impact: a share of wrong answers, blocked users. It stays low until someone notices, the time to detect (TTD), then for the time it takes to roll back, the time to recover (TTR). On a chart of quality over time, the damage is the area of the dip:

damage ≈ impact × (TTD + TTR)

and over a year: cost ≈ incident frequency × impact × (TTD + TTR).

Animation. Service quality drops when the incident starts and stays low until detection (TTD), then recovers during the rollback (TTR). The red area is the damage. Detecting in 1 h and rolling back in 15 min divides it by 45.

Example: the same incident lasts 48 h + 6 h when the alert is a user complaining, and 1 h + 15 min with an automatic alert and a rollback that has been rehearsed. The area is divided by 45. Neither of these two factors appears on the maturity staircase.

Automating retraining can cut the frequency of some incidents, but it creates others, and it touches neither TTD nor TTR. That is why I put logging (it shortens diagnosis) and rehearsed rollback (it shortens TTR) ahead of it.

My order, tested in the field

For a team shipping its first model, roughly:

  1. Ship something a human uses, by hand if needed. Until you know whether the model is worth running, the rest is speculation.
  2. Log served inputs and returned outputs. The cheapest capability of all, and without it you cannot diagnose anything.
  3. Make rollback real, and rehearse it. As long as it has never been run, rollback stays a plan on paper.
  4. Monitor the output distribution. Labels rarely arrive in time to track accuracy. The shape of what the model emits moves before the metrics do.
  5. Make training reproducible. Frozen data, frozen code, recorded parameters. Essential the day you have to explain a decision, and that day comes sooner than you think.
  6. Automate deployment. Now, because there is something worth deploying often.
  7. Automate retraining. Last, and as a deliberate decision. It is the step most capable of automating a mistake at scale, and the one that ends up on the poster.

Steps 2 and 3 are the ones teams skip. They are also the ones with the best ratio between effort and risk removed.

How I still use the curve

I keep it as a checklist, and I read it starting from the top.

I take the last step and ask, for each capability: what happens if we never build it? Sometimes the answer is "nothing, at our scale, this year". And those are often the capabilities the staircase would have had me build third.

Read that way, the maturity model goes back to what it really is: a fairly complete list of what can hurt you, drawn in the order that suits whoever published it. I keep the list, and I pick the order myself.