Evaluating a model before shipping it: my four-layer harness

A good test-set score guarantees nothing about production. How I build an evaluation pipeline that catches regressions before users do: behavioural tests, a regression set, a CI gate, and the calculation that tells you whether a score difference means anything.

A model can hit 94% F1 on the test set and disappoint within a week of going live. There is nothing mysterious about it: the number we optimized and the behaviour we shipped were never the same thing.

The layer that closes that gap is the evaluation pipeline. Below is the order in which I build it, kept deliberately down to earth on the how.

What a test-set score does not tell you

A test set (hold-out) is drawn from the same pool as the training data. Anything that happened after that pool was collected is missing from it, by construction.

A hold-out set is drawn from the same distribution as the training data; production traffic arrives later and may have shiftedhistorical datasetcollected up to T0trainhold-outproduction trafficarriving after T0distribution: unknownT0nowshift lives heresame distribution — measurable
Figure 1. The test set measures error inside the distribution it was drawn from. Whatever changed after T0 is out of its reach: a great offline score says nothing about next month.

Three problems hide behind a good offline score.

Distribution drift. Inputs change: new product categories, a new customer segment, a UI change that alters how people phrase their requests. The model is right about the old question, but the question has changed.

Label leakage. A feature contains the answer in a form that will not exist at inference time. An updated_at column that is only written once the outcome is known gives a spectacular test score and a useless model.

Metric and goal mismatch. F1 counts every error the same way. Production rarely does. On a fraud check, a false negative and a false positive have different costs, different escalation paths, and different people complaining.

The first two can be found offline if you look for them. The third is a design choice: settle it before picking the metric, because no evaluation will fix it afterwards.

The four layers of the harness

I think of evaluation as four layers. Each one catches what the previous one cannot see.

Four evaluation layers — unit tests, behavioural suites, regression sets, online metrics — with production incidents feeding back into the regression set1 · unit testssingle input, asserted output — runs in milliseconds2 · behavioural suitesgroups of inputs probing one capability3 · regression setevery past production failure, kept forever4 · online metricswhat actually happens once deployedincidentsomething brokebecomes a rowCI gate runs 1–3 on every commit
Figure 2. Layers 1 to 3 run in CI and answer "did we break something we already fixed?". Layer 4 is the only one that measures reality, and its job is to keep feeding layer 3.

Layer 1, unit tests on the model. They cover the model's behaviour, and the training code has its own tests alongside. A known input, a checked output. They cost nothing, run on every commit and catch the silly regressions: a preprocessing step that lowercases what it should not, a tokenizer change that truncates without warning.

Layer 2, behavioural suites. Groups of inputs that test a capability. "Does the classifier still handle negation?" takes twenty sentences, one assertion is not enough. The benefit: when they fail, you know which capability broke, on top of knowing that the score moved.

Layer 3, the regression set. Every bug found in production becomes a permanent row. It is the layer teams skip, and the one that gains value over time: after a year it holds everything the system ever got wrong.

Layer 4, online metrics. The only layer that measures what really happens. Everything above remains an indirect signal.

My regression set fits in a CSV

The format has to be simple enough that adding a row in the middle of an incident is not a chore.

class=class="tok-str">"tok-com"># regressions.csv — id,input,expected,note,added_at,ticket
class=class="tok-str">"tok-com">#
class=class="tok-str">"tok-com"># One row per production failure. Never deleted, even long after the bug
class=class="tok-str">"tok-com"># is fixed: that row is what keeps it from coming back.

import csv
from pathlib import Path

import pytest

ROWS = list(csv.DictReader(Path("tests/regressions.csv").open()))

@pytest.mark.parametrize("row", ROWS, ids=lambda r: r["id"])
def test_regression(row, model):
    got = model.predict(row["input"])
    assert got == row["expected"], (
        f"regression {row[&#class="tok-num">39;id&#class="tok-num">39;]} ({row[&#class="tok-num">39;ticket&#class="tok-num">39;]}): {row[&#class="tok-num">39;note&#class="tok-num">39;]}\n"
        f"  expected {row[&#class="tok-num">39;expected&#class="tok-num">39;]!r}, got {got!r}"
    )

The note and ticket columns matter more than they look. Eighteen months later, a failing row with no context looks like a wrong expectation, and someone will end up "fixing" it by changing the expected value.

Deterministic first, exhaustive later

An evaluation suite that gives two different results on two back-to-back runs does more harm than good: the team learns to ignore red builds.

  • Pin everything that has a version. Model weights by hash (a tag can move), the tokenizer, the prompt template and the library that renders it. A prompt template is code: it goes in Git.
  • Fix the sampling parameters. temperature=0 is necessary, and it is not always enough: on some kernels, the order of floating-point reductions depends on batch composition. For bit-identical output, pin the batch size too.
  • Freeze the data. I address the evaluation set by its content (a hash). If the regression set grew between two runs, the comparison means nothing.
  • Separate flakiness from failure. Before trusting a red build, I run the suite a second time on the same commit. If the two runs disagree, it is a determinism bug, and it comes before anything else.

Judging open-ended output

Exact match works for classification and stops working as soon as the output is a sentence. The progression I follow, from cheapest to most expensive:

Structural assertions. Before judging quality, I check what costs nothing: valid JSON, required fields present, no leaked prompt fragments, length within bounds, citations pointing to documents that exist. A good share of real failures gets caught here, deterministically and for free.

Rubric scoring. I write the rubric before looking at outputs. A rubric written afterwards describes what the model already does.

Model-as-judge, with its known biases. Useful, but its flaws are documented: position bias when comparing two answers, a preference for longer answers, a preference for its own model family. The countermeasures: shuffle the order and run both directions, strip formatting before judging, use a judge from a different family than the system under test.

Pairwise comparison. "Is A better than B?" gives much more stable answers than "rate A out of 10", for humans and models alike.

Recalibrate against humans, regularly. I take 50 items, have a person score them blind, and measure agreement with the automatic judge. If agreement drifts, the judge has changed, and the model under test has nothing to do with it.

Wiring it into CI

The gate matters as much as the tests. A suite that cannot block a merge is documentation.

class=class="tok-str">"tok-com"># tests/test_quality_gate.py
class=class="tok-str">"tok-com">#
class=class="tok-str">"tok-com"># Two thresholds of a different nature:
class=class="tok-str">"tok-com">#   - regression rows: ZERO tolerance. A known bug coming back blocks.
class=class="tok-str">"tok-com">#   - aggregate score: compared with the production model, with a small
class=class="tok-str">"tok-com">#     margin for noise(an absolute threshold goes stale fast).

MAX_ABSOLUTE_DROP = class="tok-num">0.02

def test_no_regression_rows_fail(results):
    failed = [r for r in results.regressions if not r.passed]
    assert not failed, f"{len(failed)} known bug(s) came back: {[r.id for r in failed]}"

def test_no_aggregate_drop(results, production_baseline):
    delta = results.score - production_baseline.score
    assert delta > -MAX_ABSOLUTE_DROP, (
        f"score dropped {abs(delta):.3f} vs production "
        f"({production_baseline.version}); threshold {MAX_ABSOLUTE_DROP}"
    )

Two details decide whether it holds up over time.

Compare with production. An absolute threshold is right for about a week. Comparing with the model currently deployed keeps the gate relevant as the system improves, and surfaces cases like "we improved everywhere except on this capability".

Publish the diff. A CI comment listing which suites moved, and in which direction, can be read and acted on. A single score on its own says very little.

The maths, intuitively: the noise in a score

A score measured on an evaluation set is a measurement made on a sample. Draw 100 other examples from the same pool and the same model will get 87%, then 92%, then 89%. The 2-point threshold of the gate above only makes sense if that noise is smaller than it.

The size of that noise can be computed. For a score p (the share of correct answers) measured on n examples, the standard deviation is:

σ = √( p(1 − p) / n )

and 95% of measurements fall within ± 1.96 σ of the true score. The point that matters is the n under the square root: to halve the noise you need four times as many examples.

Animation. 100 examples drawn at random, correct (mint) or wrong (red). Each draw gives a different score, and 95% of them fall inside the band. With 400 examples the band is half as wide.

To give an idea: at 90% on 500 examples, σ ≈ 1.3 points, so an interval of ± 2.6 points. A 2-point gate on such a set will sometimes fire for nothing. Three countermeasures, from cheapest to most expensive:

  • compare both models on the same examples: the noise from picking examples becomes shared and largely cancels out;
  • look at the examples whose verdict changed, on top of the score difference;
  • grow the set, keeping in mind that precision costs the square of what it brings.

Closing the loop

The loop that matters is short: production reveals a failure, the failure becomes a row, the row runs on every commit. The rest of this article is tooling around that loop.

A simple test to see where you stand: can you reproduce locally, in under five minutes, the regression seen in production yesterday? If not, you have a test suite, and the evaluation pipeline is still missing.

Where to start

For a team shipping its first model, in this order:

  1. Structural assertions on the output. An afternoon of work, and the embarrassing failures are caught.
  2. A regression CSV and the ten-line test that reads it. Start it with the first incident.
  3. A comparison with the deployed model, in CI. The threshold is computed relative to it.
  4. Behavioural suites, once you know which capabilities really matter. Nobody guesses them right on day one, and that is fine.

Online metrics come last in build order and first in importance. It is uncomfortable, but you cannot instrument what you have not shipped yet.