vLLM, TGI, SGLang — who wins at batch=1?

Throughput benchmarks answer the wrong question for an assistant. Here is the single-stream protocol, the metrics that matter, and the measurement traps that invalidate most published numbers.

Most published LLM serving benchmarks optimise for throughput — batch size 32, 64, 128, tokens per second aggregated across concurrent requests. That is the right metric for a search index or a bulk classification job.

It is the wrong metric for an assistant. One user sends one prompt and waits. Batch size is one, the GPU is mostly idle, and every optimisation that trades latency for occupancy is working against you.

This piece is the protocol for measuring that case properly, and the reasons most single-stream numbers you will find are not comparable.

The anatomy of one request

Before choosing an engine, agree on what is being measured. A single streamed request has three distinct phases, and engines differ in which phase they optimise.

Figure 1 — Prefill is compute-bound and scales with prompt length; decode is memory-bandwidth-bound and scales with the KV cache. An engine can be excellent at one and mediocre at the other, which is why a single "tokens/sec" number hides the answer.

Three numbers, three different things:

TTFT — how long before anything appears. This is what a user experiences as "did it hear me?". Dominated by prefill, so it scales with prompt length.
ITL / TPOT — the gap between successive tokens. This is what a user experiences as reading speed. Dominated by memory bandwidth, so it scales with KV cache size and model width, not prompt length.
End-to-end — the sum. Useful for capacity planning, misleading for UX, because a fast TTFT with slow decode feels better than the reverse at identical totals.

At batch=1, prefill leaves most of the GPU idle: there is no other request to fill the gaps. That is precisely why throughput-oriented optimisations do not transfer.

Protocol

Hardware: 1× H100 SXM, 80GB, PCIe Gen5
Models: Mistral-7B, Llama-3-8B, Qwen2-7B, Phi-3-Medium, Gemma-7B, Zephyr-7B-β
Engines: vLLM 0.5.0, TGI 2.0.4, SGLang 0.2.5
Workload: 2048 prompts, ShareGPT distribution, ctx=4k, max_new=256
Concurrency: 1, 4, 16, 64
Metrics: TTFT, ITL, end-to-end p50/p99

Each run repeated 3 times; figures below are medians of medians.

TL;DR

At batch=1, the ranking is not the one throughput benchmarks suggest:

SGLang leads on TTFT for prompt-heavy workloads, thanks to RadixAttention prefix reuse
vLLM leads on ITL once decoding starts
TGI is consistent — never fastest, never slowest

Time to first token (lower is better)

Single-stream TTFT, prompts of ~800 tokens:

Engine	Mistral-7B	Llama-3-8B	Qwen2-7B	Phi-3-M
vLLM	82ms	96ms	79ms	71ms
TGI	94ms	104ms	91ms	84ms
SGLang	68ms	82ms	65ms	58ms

The SGLang advantage shrinks on cold caches. Most assistants carry a long, stable system prompt, so in practice the cache is warm and the advantage is real — but it is an advantage on your prompt shape, not a universal one.

Why the decode phase diverges

Once decoding starts, the bottleneck moves from compute to memory bandwidth: every token requires reading the entire KV cache. How an engine lays that cache out therefore decides decode speed.

PagedAttention (vLLM) stores the KV cache in fixed-size blocks, the way an OS pages memory. It nearly eliminates the internal fragmentation of a contiguous-allocation scheme, so more of the cache fits in fewer, denser reads.

RadixAttention (SGLang) organises cached prefixes in a radix tree, so requests sharing a prefix share its KV entries. For a system prompt repeated across every request, the prefill for that shared span happens once rather than once per request — which is exactly the assistant workload.

These optimise different phases, which is why the ranking flips between the TTFT table and the decode numbers. The two techniques are not mutually exclusive and the gap between engines has been closing steadily; treat any specific ranking as perishable.

The traps that invalidate most published numbers

This is the part worth more than the table.

Prefix cache state is not reported. A warm prefix cache can cut TTFT dramatically for a repeated system prompt. A benchmark that does not state whether the cache was warm, cold, or reset between runs is not comparable to anything. Report both, separately.

Client overhead counted as server latency. Measuring with a Python client that does JSON parsing per streamed chunk can add milliseconds of the same order as the difference you are trying to detect. Measure server-side where possible, and always report which side of the wire the clock is on.

Tokeniser differences, uncompared. Two engines given "the same prompt" may produce different token counts for the same string. Per-token metrics then compare different quantities. Fix the comparison by tokenising once, up front, and feeding token IDs — or at minimum report the token counts alongside the latencies.

Streaming chunk granularity. Some servers emit one token per SSE event, others batch several. If your ITL is derived from event arrival times, you are measuring the server's flush policy, not its decode speed.

GPU clocks are not constant. Sustained load pushes the card into thermal or power limits, and an unlocked boost clock drifts over a long run. Lock clocks for the duration of the measurement, and discard warmup:

"tok-com"># Lock the clocks so the first and last runs are comparable.
sudo nvidia-smi -pm 1
sudo nvidia-smi -lgc 1980,1980          # pick a value the card sustains
nvidia-smi --query-gpu=clocks.sm,temperature.gpu,power.draw \
           --format=csv -l 5 > clocks.csv   # keep this next to the results

One run is noise. Single-stream latency has a long right tail from allocator behaviour and scheduler jitter. Three repetitions is a floor, not a target, and the median of medians is more honest than a mean.

Versions move fast. Every number in this article is attached to a specific engine version. Inference engines change more between minor releases than most libraries do between majors.

Reproducing

"tok-com"># vLLM, single-stream, prefix caching on.
docker run --gpus all --shm-size=8g -p 8000:8000 \
  vllm/vllm-openai:v0.5.0 \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --max-model-len 8192 \
  --enable-prefix-caching
"tok-com">
# Warm the prefix cache, then measure. The two runs are reported separately:
"tok-com"># merging them produces a number that describes neither case.
python bench.py --concurrency 1 --warmup 32 --repeat 3 --cache-state cold
python bench.py --concurrency 1 --warmup 32 --repeat 3 --cache-state warm

What this means when choosing

If your workload is a chat assistant with a long shared system prompt, prefix reuse is likely to dominate everything else, and you should measure TTFT warm and cold before reading any other comparison.

If your workload is long-form generation from short prompts, decode speed dominates and prefix reuse is close to irrelevant.

If you cannot tell which you are, that is the measurement to run first — on your own prompt distribution. A benchmark on ShareGPT tells you about ShareGPT.

Caveats

Single hardware configuration, single set of engine versions, one prompt distribution. The protocol transfers; the ranking does not. Run it on your own traffic before committing to an engine — the point of publishing a protocol is that you do not have to trust the table.