Monitoring an LLM from day one: latency, drift, quality, cost

Four signals to track on an LLM service from the day it goes live, why classic application monitoring misses them, and the order in which I instrument them.

Classic application monitoring answers two questions: is it up, and is it fast. On an LLM service both can be green while the service has become useless: the endpoint returns 200s in 400 ms, with answers that have been wrong for three days.

Four signals fill that gap. None of them is exotic. They get forgotten because only the first one looks like monitoring.

Four observability signals for an LLM service — latency, drift, semantic quality, cost — and what each one catches1 · latencyTTFT, ITL, e2equeue depthcatches:capacity problems2 · driftinput embeddingsoutput length, refusalscatches:the world changing3 · semanticsschema validitycitation resolutioncatches:quality regressions4 · costtokens per requestcost per outcomecatches:silent budget driftcheap to instrumentusually instrumented lasta 200 OK tells you about none of these
Figure 1. Only the first signal looks like the usual monitoring. That is why the other three usually arrive after the first incident.

1. Latency, phase by phase

Overall response time hides the two things users actually feel. I record them separately:

  • TTFT (time to first token). It mostly depends on prompt processing: it varies with prompt length and cache state, and barely with answer length.
  • ITL (inter-token latency). It mostly depends on memory bandwidth: it varies with load and context length.
  • End-to-end time, with queue time as its own series.

Queue time is the one to isolate. When a service saturates, end-to-end time goes up while TTFT and ITL stay flat: the model is just as fast, requests are simply waiting their turn. Without that split, it looks like a model problem and you search in the wrong place.

On my setup, vLLM already exposes these measurements in Prometheus format and I read them in Grafana. What is left is to display them as percentiles: an average over a long-tailed distribution describes the real experience very badly.

The maths, intuitively: mean versus percentiles

The mean adds up all durations and divides by their count. Each request therefore weighs in proportion to its duration: one 30-second request weighs as much as 75 requests of 400 ms. A percentile works differently: sort the durations from shortest to longest and read a position. Out of 100 requests, p50 (the median) is the 50th, p99 the 99th. An extreme value takes a single place in that ranking, however long it is.

Animation. 100 requests sorted by duration. p50, p95 and p99 are positions in that ranking, and the mean (dotted) is a sum divided. One extra request at 30 s moves the mean and leaves the median where it was.

An example: 100 requests, a median of 0.41 s and a mean of 0.44 s. The two numbers look alike. A single extra request at 30 s pushes the mean to 0.73 s, 67% higher, while the median stays at 0.41 s. That new number describes almost nobody: two requests out of three took less than 0.5 s, no other went over 1 s, one took 30 s.

Hence my rule: p50 for the typical case, p95 and p99 for the tail, and the mean only for things that really add up, like total cost or GPU time consumed.

2. Drift, on inputs and outputs

Inputs change without warning: a new kind of user, a UI change that alters phrasing, a competitor outage that sends you unusual traffic.

On inputs. I compute embeddings for a sample of requests and track the distance between the current window and a fixed reference window. The absolute value says little, the movement is what counts. I alert on a change that lasts: alerting on spikes means getting woken up every Monday by weekly seasonality.

On outputs, the cheapest part. I track answer length, refusal rate, empty-answer rate and format-validation failure rate. No embeddings, no reference set, no extra model: these are counters, and they move before the subtler signals do.

A jump in refusal rate is one of the most telling alerts on an LLM service. It usually means a prompt changed, a model version changed, or inputs moved into territory that triggers safety training. Three things you want to know within the hour.

3. Answer quality

This is the hardest signal, and the one where most time gets lost on sophistication before the basics are in place.

Start with the deterministic part. Before judging quality, I check structure: valid JSON, required fields present, no leaked prompt fragments, citations pointing to documents that exist, length within bounds. These are ordinary assertions and they are free. In a RAG system, a broken citation is both easy to detect and serious.

Then sample and score. A small continuous sample, scored against a rubric, by a model, a human, or both. Regularity matters more than volume: 50 items a day draw a trend, 2,000 items once draw nothing.

Know the judge's biases. A model-as-judge has documented flaws: position bias when comparing two answers, a preference for longer answers, a preference for its own model family. I shuffle the order, run both directions, and use a judge from a different family than the system under evaluation.

Recalibrate the judge against humans. If automatic scores drift while agreement with humans drops, the judge has moved. Without this check you end up calling "model regression" what is really a change in the judge's behaviour.

Log user-side signals. Regenerations, edits, copies, abandons, thumbs. Noisy one by one, very clear in aggregate, and they measure the one thing that counts in the end.

4. Cost, measured against outcomes

Token counts are available for free and almost always tracked too late.

I track input and output tokens per request, cost per request, and above all cost per successful outcome: per resolved ticket, per accepted suggestion, per successful extraction. Cost per request pushes towards cheap answers. Cost per outcome pushes towards answers that are useful.

Then two drifts to watch.

Context bloat. A prompt template gains a few hundred tokens with each iteration. Every addition is defensible, and overall the bill doubles. I check the prompt token count in CI, with a hard ceiling that I raise deliberately.

Cascading retries. A validation failure triggers a retry, which fails, which retries. Latency looks fine because the successful path is fast, and cost triples. I track retries per request as a separate series and alert on the ratio.

The order I set them up in

From least to most sophisticated:

  1. Tokens and cost per request: an afternoon, useful right away.
  2. Structural validity of outputs: counters, no model, very telling.
  3. Latency split by phase: the server has to expose it (vLLM does).
  4. Refusal and length distributions: free once outputs are logged.
  5. Input embedding drift: needs a reference window and a scheduled job.
  6. Sampled quality scoring: needs a rubric, a judge and calibration.

Many teams start with 6, because it is the most interesting, and get to 1 after the first surprise bill.

The prerequisite

Everything rests on one habit: log the full request and response, with a request ID, the prompt template version and the model version. Sampled if volume requires it, but complete on the requests you keep: with truncated logs, post-incident analysis gets stuck.

Without that log, you notice that quality dropped. With it, you find which prompt version, which model version and which inputs are involved, and you move from noticing to fixing.