vLLM, TGI, SGLang at batch=1: how I measure latency

Throughput benchmarks answer a different question than an assistant, where a single user waits for an answer. The single-stream protocol I use, the metrics to track, the physical floor set by memory, and the pitfalls that make most published numbers impossible to compare.

Most published LLM serving benchmarks measure throughput: batch size 32, 64 or 128, tokens per second summed over concurrent requests. That is the right metric for indexing a corpus or classifying documents in bulk.

An assistant is a different story. One user sends a prompt and waits. Batch size is one, the GPU is nearly idle, and every optimisation that trades latency for utilisation works against you.

That is the case for my Quant project: the LLM runs with vLLM on a g6 instance (an NVIDIA L4), and I am its only user. This article covers how I measure that case, and why most single-stream numbers you find online cannot be compared with each other.

Anatomy of a request

Before picking an engine, you need to know what you are measuring. A streaming request has three phases, and engines do not all optimise the same one.

Timeline of a single streamed request, showing queue, prefill, first token, and the decode steps that followqueueprefill (prompt)tok 1tok 2tok 3···tok nTTFT — time to first tokenITL — inter-token latencyend-to-endcompute-bound ↑ prefill · memory-bandwidth-bound ↑ decode
Figure 1. Prefill is compute-bound and grows with prompt length. Decode is memory-bandwidth-bound and grows with the KV cache. An engine can be excellent at one and average at the other: a single "tokens/s" number hides all of that.

Three numbers, three different things:

  • TTFT: the delay before the first token appears. It is what users feel as "did it hear me?". It mostly depends on prefill, so on prompt length.
  • ITL / TPOT: the interval between two tokens. It is the perceived reading speed. It mostly depends on memory bandwidth, KV cache size and model size, and barely on prompt length.
  • End-to-end time: the sum. Useful for sizing, misleading for experience: for the same total time, a fast first token followed by slow decoding feels better than the reverse.

At batch=1, prefill leaves most of the GPU idle, since no other request fills the gaps. That is why optimisations designed for throughput do not carry over.

The protocol

  • Hardware: a single GPU, always the same one, with locked clocks (see below).
  • Models: models of roughly 7 to 8 billion parameters, served in 16-bit.
  • Engines: one specific version of each engine, recorded with the results.
  • Workload: a sample of prompts representative of real traffic, 4k context, at most 256 output tokens.
  • Concurrency: 1, then 4, 16 and 64 to see where latency breaks down.
  • Metrics: TTFT, ITL and end-to-end, at p50 and p99.

Each series runs at least 3 times, and I report the median of medians.

Why engines diverge during decoding

Once decoding starts, the bottleneck moves from compute to memory bandwidth: every token requires reading the whole KV cache again. How an engine lays out that cache therefore decides its decoding speed.

PagedAttention (vLLM) stores the KV cache in fixed-size blocks, the way an operating system pages memory. The internal fragmentation of a contiguous allocation nearly disappears: more of the cache fits into fewer, denser reads.

RadixAttention (SGLang) organises cached prefixes in a radix tree: requests that share a prefix share its KV entries. For a system prompt repeated in every request, the prefill of that shared segment happens once instead of once per request. That is exactly an assistant's workload.

The two techniques target different phases. So one engine can lead on warm-cache TTFT while another leads on decoding speed. They can be combined, and the gap between engines narrows with each release: any precise ranking goes stale fast.

As for TGI, its official documentation says the project is now in maintenance mode: only minor fixes are still accepted, and Hugging Face points to vLLM, SGLang, or local engines such as llama.cpp and MLX. It changes nothing about the measurement method, but it is worth weighing before building on it.

The maths, intuitively: memory sets the speed

To produce one token, the GPU has to read all of the model's weights from its memory (HBM), plus the KV cache. At batch=1 it has very little computing to do with what it reads: it spends most of its time waiting for data. The time between two tokens therefore has a floor:

time per token ≥ size of the weights / memory bandwidth

Animation. For every token, the 14.5 GB of weights cross the memory bus while compute stays almost idle. At batch=32, the same read produces 32 tokens.

A 7-billion-parameter model in 16-bit weighs about 14.5 GB. An H100 SXM reads its memory at about 3.35 TB/s, giving a floor of about 4.3 ms per token, so at most ~230 tokens per second for a single user, whatever the engine. On my L4, with a bandwidth of about 300 GB/s, the same calculation gives a floor of about 48 ms per token, around twenty tokens per second at best. Engines do not go below that floor, they get more or less close to it: fewer wasted KV cache reads with PagedAttention, less repeated prefill with RadixAttention.

At batch=32, the same read of the weights serves 32 requests at once. Total throughput is multiplied, but each user still waits at least those few milliseconds between two tokens. That is why throughput benchmarks say nothing about the batch=1 experience, and why 8-bit or 4-bit quantisation speeds up decoding: it cuts the number of bytes to read, without touching the compute.

The pitfalls that make numbers incomparable

This is the part of the article I use the most.

Prefix cache state is not reported. A warm prefix cache can cut TTFT sharply when the system prompt repeats. A benchmark that does not say whether the cache was warm, cold or flushed between series compares to nothing. I report both cases, separately.

Client time counted as server latency. A Python client parsing JSON on every received chunk can add a few milliseconds, the same order as the gap you are trying to measure. I measure server-side when possible, and always state which side of the network the clock sits on.

Different tokenizers. Two engines given "the same prompt" can count a different number of tokens for the same string. Per-token metrics then compare different quantities. The fix: tokenize once upstream and send token IDs, or at least publish token counts alongside latencies.

Streaming chunk size. Some servers emit one token per SSE event, others batch several. An ITL computed from event arrival times measures the server's flushing policy, and its decoding speed stays unknown.

GPU clocks move. Under sustained load the card hits its thermal or power limits, and an unlocked boost clock drifts over the series. I lock the clocks during measurement and discard the warm-up:

"tok-com"># Lock clocks so the first and last series are comparable.
sudo nvidia-smi -pm 1
sudo nvidia-smi -lgc 1980,1980          # a value the card can sustain(adapt to your GPU)
nvidia-smi --query-gpu=clocks.sm,temperature.gpu,power.draw \
           --format=csv -l 5 > clocks.csv   # keep it next to the results

One series is noise. Single-stream latency has a long right tail, from the allocator and scheduler jitter. Three repetitions are a minimum, and the median of medians reflects reality better than a mean.

Versions move fast. Every number is tied to a specific engine version. Inference engines change more between two minor versions than most libraries do between two major ones.

Reproducing

"tok-com"># vLLM, single stream, prefix caching on. Pinned version: record it with the results.
VLLM_VERSION=<tested version>
docker run --gpus all --shm-size=8g -p 8000:8000 \
  vllm/vllm-openai:${VLLM_VERSION} \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --max-model-len 8192 \
  --enable-prefix-caching

Then two series at concurrency 1 with the same prompt set: one with a cold cache (server restarted), one with a warm cache (after about thirty warm-up requests). I report them separately: merging them gives a number that describes neither case.

When it is time to choose

If the workload is a chat assistant with a long shared system prompt, prefix reuse will probably outweigh everything else. I then measure TTFT warm and cold before reading any other comparison.

If the workload is long generation from short prompts, decoding speed is what counts, and prefix reuse brings almost nothing.

If you do not know which case you are in, that is the first measurement to make, on your own prompt distribution. A benchmark run on ShareGPT tells you about ShareGPT.

Limits

The protocol carries over from one machine to another, rankings much less so. A result holds for one GPU, one set of engine versions and one prompt distribution. That is the whole point of publishing the protocol: anyone can run it on their own traffic before choosing an engine.