What happens between an LLM request and the answer on screen?

Last reviewed: 2026-10 · Prerequisite: KV cache

A model answers quickly on its own, then feels slow behind a website. The forward pass may not be the problem. Requests can wait in a queue, CPU input processing can fall behind, and a newly arrived document can occupy the next batch.

Follow one request through the service, then use a small scheduling example to see why higher throughput and smoother streaming are not always the same goal. We focus on ordinary autoregressive generation. Speculative decoding, hybrid-state models, and cross-machine caches add other paths.

Start with who does what

vLLM V1 separates API handling, engine-core scheduling and KV management, and GPU execution. These are responsibilities, not a claim that every deployment runs exactly three processes. Official architecture overview

Trace a short request

Suppose the prepared prompt is [A, B, C] and the generated answer is [D, E]. Letters stand in for token IDs.

The first generated token is sampled too. Prefill, decode, and sampling are not three stages that each run once. Whenever the model selects another token, it applies the relevant logit-processing and sampling rules. An intermediate prefill chunk usually cannot produce an answer yet; the request must reach a generation boundary.

Output processing is more than converting each token ID into one character. A character or stop string can span tokens, requiring buffering. “A token was computed” and “a text chunk is ready to send” are different events. Follow AsyncLLM.generate and the output processor at the pinned version to connect the two.

A long prompt arrives while others are streaming

Two requests are decoding, each needing one input position this iteration. A new 10-token prompt arrives. To isolate chunking, allow six input positions per iteration, schedule decode first, and give the remaining budget to prefill.

The new request can produce its first token after completing prefill in iteration 3. Existing requests receive compute each iteration. With a budget of 12, the prompt fits in one iteration—but that iteration may take longer, pausing the streams already in progress.

This is budget arithmetic, not a latency prediction. One prefill position and one decode position need not cost the same. KV capacity, batch shapes, priorities, and kernels also matter. Six scheduled positions do not imply a fixed number of milliseconds. Why a larger decode batch moves from bandwidth-bound toward compute-bound is covered in Performance: decode batching.

def prefill_chunks(prompt_tokens, decode_tokens, token_budget):
    if any(type(value) is not int for value in
           (prompt_tokens, decode_tokens, token_budget)):
        raise ValueError("Token counts must be integers")
    if prompt_tokens < 0 or not 0 <= decode_tokens <= token_budget or token_budget < 1:
        raise ValueError("Invalid token budget")
    remaining_budget = token_budget - decode_tokens
    if prompt_tokens and remaining_budget <= 0:
        raise ValueError("No capacity left for prefill in this toy schedule")
    chunks = []
    while prompt_tokens:
        scheduled = min(prompt_tokens, remaining_budget)
        chunks.append(scheduled)
        prompt_tokens -= scheduled
    return chunks

assert prefill_chunks(10, 2, 6) == [4, 4, 2]
assert prefill_chunks(10, 2, 12) == [10]

The vLLM tuning guide discusses the TTFT / ITL tradeoff. The current V1 scheduler does not maintain two entirely separate prefill and decode phases: it tracks computed positions and assigns additional work. Our table explains the budget without pretending to reproduce that implementation.

Cancellation has to reach the engine

A user asks for up to 50 tokens but closes the page after five. If the service computes the rest anyway, it consumes GPU time and cache capacity for output nobody will receive.

Both normal completion and cancellation need cleanup: stop accepting further output for the request, notify the engine, stop scheduling new work, and release cache references. Already-issued GPU work may not be interruptible at an arbitrary instruction, so cancellation needs a defined completion boundary. At the pinned version, generate handles cancellation by aborting the request; the engine iteration also processes aborts received during execution.

Releasing a reference does not return memory to the operating system. Another request may still use the prefix, or an unused cached block may remain in a preallocated pool until reuse or eviction. Continue with the cache lifecycle for that distinction.

Measure the delay before tuning parameters

For the first token, instrument input processing, queueing, the model computation needed for first output, and output handling / transport. These are useful measurement boundaries, not necessarily disjoint intervals. Adding separate mean durations does not give end-to-end P95 latency.

Load generation matters too. In a fixed-concurrency test, a completed request is replaced by another; a slower service also receives replacements more slowly. With a fixed arrival rate, queues can keep growing. Both tests are useful, but their tail latencies are not directly interchangeable. Record lengths, arrival pattern, cold or warm caches, hardware, and versions. Report failures and timeouts separately.

Use the flow to choose a deployment

Fitting model weights on one GPU does not establish that the GPU can serve your target concurrency. A complete memory budget includes weights, KV, and runtime headroom. Identify the resource you are short of before scaling. What independent replicas, TP, PP and prefill/decode separation each help with and cost, and how to do the weight arithmetic for eight GPUs, are in vLLM and SGLang: several ways to use several GPUs; how replicas are scheduled and scaled in a cluster is in Kubernetes and GPU clusters.

This is not a framework ranking. Start with a simple working deployment, measure quality, cost, and latency on representative traffic, and then decide whether another system layer is worth it. Parallelism mechanics explains partitioning; inference capacity and service behavior still need their own validation.

Where to enter the source code

This note checks vLLM 10cc2f6; source links use the full commit rather than moving main. Start with the completion API handler, then follow AsyncLLM.generate → EngineCore.step → Scheduler.schedule → worker execution / sampling → output processor. Chat and speculative decoding introduce branches. Async scheduling is on by default for compatible configurations at this commit (AsyncScheduler subclasses Scheduler), so the CPU prepares the next step while the GPU runs the current one, and the two steps can overlap in a trace. Some pooling models, executors, or speculative-decoding configurations disable it. Not every request follows one identical call stack.

We run the budget example and documentation checks, not a complete serving deployment of this version. No framework speedup is claimed. Reproducing performance also requires fixed models, hardware, engine settings, and request replay.