Observability for LLMs: latency, drift, semantics, cost

Four signals every LLM service needs from day one, why the usual application monitoring misses all of them, and the one that is almost always instrumented last.

Standard application monitoring answers "is it up" and "is it fast". For an LLM service, both can be green while the thing is quietly useless: the endpoint returns 200s, in 400ms, with answers that stopped being correct three days ago.

Four signals cover the gap. None of them is exotic; the reason they get skipped is that only the first looks like monitoring.

Four observability signals for an LLM service — latency, drift, semantic quality, cost — and what each one catches1 · latencyTTFT, ITL, e2equeue depthcatches:capacity problems2 · driftinput embeddingsoutput length, refusalscatches:the world changing3 · semanticsschema validitycitation resolutioncatches:quality regressions4 · costtokens per requestcost per outcomecatches:silent budget driftcheap to instrumentusually instrumented lasta 200 OK tells you about none of these
Figure 1 — Only the first signal resembles conventional monitoring, which is why the other three tend to be added after the first incident rather than before it.

1. Latency, split by phase

Aggregate response time hides the two things a user actually feels. Record them separately:

Queue wait is the one worth isolating. When a service saturates, end-to-end latency rises while TTFT and ITL stay flat — the model is as fast as ever, requests are simply waiting. Without the split, that looks like a model problem and gets debugged as one.

Percentiles, not means. A mean over a long-tailed distribution is a number that describes nobody.

2. Drift, on inputs and outputs

Inputs change without warning: a new customer segment, a UI change that alters phrasing, a competitor's outage sending you unfamiliar traffic.

On inputs: embed a sample of requests and track the distance between the current window and a fixed reference window. The absolute value means little; the movement is the signal. Alert on a sustained change, not on a spike — daily and weekly seasonality will otherwise page you every Monday.

On outputs, and this is the cheap half: track response length, refusal rate, empty-response rate, and the rate of failed format validation. These need no embeddings, no reference set and no extra model. They are counters, and they move before anything subtle does.

A refusal-rate jump is one of the highest-signal alerts in an LLM service. It usually means a prompt change, a model version change, or an input distribution that has wandered somewhere the safety training reacts to — all three are things you want to know about within the hour.

3. Semantic quality

The hard one, and the one where most effort is wasted on sophistication before the basics are in place.

Start with what is deterministic. Before judging quality, assert structure: valid JSON, required fields, no leaked prompt fragments, citations resolving to documents that exist, length within bounds. These are ordinary assertions, they cost nothing, and in a retrieval-augmented system a broken citation is both detectable and serious.

Then sample and score. A small, continuous sample scored by a rubric — by a model, by a human, or both. Continuous matters more than large: 50 items a day trends; 2000 items once does not.

Know the judge's biases. Model-as-judge has documented failure modes: position bias in pairwise comparisons, preference for longer answers, and self-preference when grading its own model family. Randomise order, run both directions, and prefer a judge from a different family than the system under test.

Calibrate the judge against humans, periodically. If automated scores drift while human agreement drops, the judge changed, not the system. Without this check you eventually alert on the judge's behaviour and call it a model regression.

Log user-visible signals. Regenerations, edits, copy events, abandonment, thumbs. Noisy individually, unambiguous in aggregate, and they measure the only thing that actually matters.

4. Cost, per outcome

Token counts are trivially available and almost always tracked too late.

Track input and output tokens per request, cost per request, and — the one that changes decisions — cost per completed outcome: per resolved ticket, per accepted suggestion, per successful extraction. Cost per request optimises for cheap answers; cost per outcome optimises for answers that work.

Then watch for the two silent drifts:

Context creep. A prompt template grows by a few hundred tokens per iteration. Each change is defensible, the cumulative effect is a doubled bill. Assert on prompt token count in CI — a hard ceiling, reviewed when it is raised.

Retry amplification. A validation failure triggers a retry, which fails, which retries. Latency looks acceptable because the successful path is fast; cost triples. Track retries per request as its own series, and alert on the ratio rather than the count.

The ordering nobody follows

Instrument in reverse order of sophistication:

  1. Tokens and cost per request — one afternoon, immediately actionable
  2. Structural validity of outputs — counters, no model, high signal
  3. Latency split by phase — requires the server to expose it
  4. Refusal and length distributions — free once outputs are logged
  5. Input embedding drift — needs a reference window and a scheduled job
  6. Sampled semantic scoring — needs a rubric, a judge and calibration

Most teams start at 6, because it is the interesting one, and reach 1 after the first surprising invoice.

The prerequisite

All of it rests on one practice: log the full request and response, with a request id, a prompt template version and a model version. Sampled if volume demands it, but complete on the sampled requests — truncated logs are the reason post-incident analysis stalls.

Without that log you can observe that quality dropped. With it, you can find out which prompt version, which model version and which inputs — which is the difference between an incident report and a fix.