Shipping a model you trust — a field guide to eval pipelines

Why a hold-out score is not a promise about production, and how to build an eval harness that catches regressions before your users do.

A model can score 94% F1 on a hold-out set and still fail within a week of going live. That is not a paradox and not bad luck — it is what happens when the number you optimised for and the behaviour you shipped were never the same thing.

This is a field guide to the layer that catches that gap: the eval pipeline. It is deliberately opinionated about what to build first, and deliberately boring about how.

Why a hold-out score is not a promise

A hold-out set is drawn from the same pool as the training data. Whatever the world did after that pool was collected is, by construction, absent from it.

Figure 1 — The hold-out set can only measure error within the distribution it was sampled from. Everything that changed after T0 is outside its reach, which is why a strong offline score says nothing about next month.

Three failure modes hide behind a good offline score:

Distribution shift. The inputs change. New product categories, a new customer segment, a UI change that alters how people phrase queries. The model is not wrong; the question is.

Label leakage. A feature encodes the answer in a way that will not exist at inference time. An updated_at column that only gets written after the outcome is known will give you a spectacular hold-out score and a useless model.

Metric–objective gap. F1 treats every error as equivalent. Production rarely does. A false negative on a fraud check and a false positive on a fraud check have different costs, different escalation paths and different people complaining.

The first two are detectable offline if you look. The third is a design error that no amount of evaluation will fix — it has to be decided before you pick a metric.

The four layers of an eval harness

Think of evaluation as four layers, each catching what the one below it cannot.

Figure 2 — Layers 1 to 3 run in CI and answer "did we break something we already fixed?". Layer 4 is the only one that tells you the truth, and its job is to keep feeding layer 3.

Layer 1 — unit tests for the model. Not for the training code: for the model's behaviour. A known input, an asserted output. These are cheap, they run on every commit, and they catch the stupid regressions — a preprocessing change that lowercases something it should not, a tokeniser swap that silently truncates.

Layer 2 — behavioural suites. Groups of inputs that probe one capability rather than one example. "Does the classifier still handle negation?" is a suite of twenty sentences, not one assertion. The value is that it fails informatively: you learn which capability broke, not just that the score moved.

Layer 3 — regression sets. Every bug found in production becomes a permanent row. This is the layer teams skip, and it is the one that compounds: after a year, it encodes everything your system has ever got wrong.

Layer 4 — online metrics. The only layer measuring reality. Everything above it is a proxy.

A regression set is a CSV, not a framework

The implementation should be boring enough that adding a row during an incident is not a chore.

class=class="tok-str">"tok-com"># regressions.csv — id,input,expected,note,added_at,ticket
class=class="tok-str">"tok-com">#
class=class="tok-str">"tok-com"># One row per production failure. Never deleted, even when the underlying
class=class="tok-str">"tok-com"># bug is long fixed: the row is what stops it coming back.

import csv
from pathlib import Path

import pytest

ROWS = list(csv.DictReader(Path("tests/regressions.csv").open()))

@pytest.mark.parametrize("row", ROWS, ids=lambda r: r["id"])
def test_regression(row, model):
    got = model.predict(row["input"])
    assert got == row["expected"], (
        f"regression {row[&#class="tok-num">39;id&#class="tok-num">39;]} ({row[&#class="tok-num">39;ticket&#class="tok-num">39;]}): {row[&#class="tok-num">39;note&#class="tok-num">39;]}\n"
        f"  expected {row[&#class="tok-num">39;expected&#class="tok-num">39;]!r}, got {got!r}"
    )

The note and ticket columns matter more than they look. Eighteen months later, a failing row with no context is indistinguishable from a bad expectation, and someone will "fix" it by editing the expected value.

Make evals deterministic before you make them comprehensive

An eval suite that gives a different answer on two consecutive runs is worse than no suite: it trains the team to ignore red builds.

Pin everything that has a version. Model weights by digest, not by tag. Tokeniser, prompt template, and the library that renders it. A prompt template is code and belongs in version control.
Fix the sampling parameters. temperature=0 is necessary and not sufficient — batching can still change results, because floating-point reduction order depends on batch composition on some kernels. If you need bit-exact output, pin the batch size too.
Freeze the data. Content-address the eval set. A regression set that quietly grew between two runs makes the comparison meaningless.
Separate flakiness from failure. Run the suite twice on the same commit before you trust a red. If the two runs disagree, you have a determinism bug, and it outranks whatever you were about to investigate.

Judging open-ended output

Exact match works for classification and falls apart the moment the output is a sentence. The usual progression, in increasing order of cost and decreasing order of self-deception:

Structural assertions first. Before judging quality, assert the cheap things: valid JSON, required fields present, no leaked prompt fragments, length within bounds, citations resolve to documents that exist. A surprising share of real failures are caught here, deterministically, for free.

Rubric-based scoring. Write the rubric before you look at the outputs. A rubric written afterwards describes what the model already does.

Model-as-judge, with its known biases. Useful, but it has documented failure modes: position bias in pairwise comparisons, a preference for longer answers, and self-preference when a model grades its own family. Mitigations: randomise the order and run both directions, strip formatting before judging, and use a judge from a different family than the system under test.

Pairwise beats absolute. "Is A better than B" is a far more stable question than "score A out of 10", for humans and models alike.

Calibrate against humans, periodically. Sample 50 items, have a person grade them blind, and measure agreement with your automated judge. If agreement drifts, the judge — not the model — is what changed.

Wiring it into CI

The gate matters as much as the tests. A suite nobody can block a merge with is documentation.

class=class="tok-str">"tok-com"># tests/test_quality_gate.py
class=class="tok-str">"tok-com">#
class=class="tok-str">"tok-com"># Two thresholds, deliberately different in kind:
class=class="tok-str">"tok-com">#   - regression rows: ZERO tolerance. A known bug coming back blocks.
class=class="tok-str">"tok-com">#   - aggregate score: relative to the current production model, with a
class=class="tok-str">"tok-com">#     small tolerance band for noise — absolute thresholds rot.

MAX_ABSOLUTE_DROP = class="tok-num">0.02

def test_no_regression_rows_fail(results):
    failed = [r for r in results.regressions if not r.passed]
    assert not failed, f"{len(failed)} known bug(s) came back: {[r.id for r in failed]}"

def test_no_aggregate_drop(results, production_baseline):
    delta = results.score - production_baseline.score
    assert delta > -MAX_ABSOLUTE_DROP, (
        f"score dropped {abs(delta):.3f} vs production "
        f"({production_baseline.version}); threshold {MAX_ABSOLUTE_DROP}"
    )

Two details that decide whether this survives contact with a team:

Compare against production, not against a constant. An absolute threshold is correct for exactly one week. Comparing against the currently deployed model keeps the gate meaningful as the system improves — and makes "we got better everywhere except this one capability" visible.

Report the diff, not the score. A CI comment listing which suites moved, in which direction, is actionable. A single number is a coin flip with extra steps.

Closing the loop

The loop that matters is short: production surfaces a failure, the failure becomes a row, the row runs on every commit forever. Everything else in this article is scaffolding around that loop.

A practical test of whether you have an eval pipeline at all: can you reproduce yesterday's production regression, locally, in under five minutes? If not, what you have is a test suite and a hope.

What to build first

In order, for a team shipping its first model:

Structural assertions on the output. One afternoon, catches the embarrassing failures.
A regression CSV and the ten-line test that reads it. Start it on the first incident.
A comparison against the deployed model in CI. Not a threshold — a comparison.
Behavioural suites, once you know which capabilities you actually care about. You will not guess this correctly on day one, and that is fine.

Online metrics come last in build order and first in importance. That inversion is uncomfortable, and it is the honest answer: you cannot instrument what you have not shipped.