Inference: from the next token to a complete request

“The model is slow” is not specific enough. The input may be long, caches may fill memory, or requests may wait for one another. Locate the delay before choosing an optimization.

Follow one request

input text → token IDs → prefill → first token
                                      ↓
                   update KV cache ← decode → sample next token
                                      ↓
                              stop and release state

This simplified autoregressive path does not prescribe a scheduler. Prefill processes the existing context; decode uses cached state to continue generation. Sampling decides which token to select, while caching decides which state to retain and read. They solve different problems.

Similar names, different interventions

Start with generation and caching below, then move to long context and efficient attention. If you already understand decoders, find the particular mechanism you need to distinguish.

Chapter contents

For the engines that serve a model, read vLLM and SGLang: separate the application, engine, and kernels, then examine batching, prefix reuse, HiCache, and benchmarking. Mechanisms live here; product integration and ownership belong in engineering practice.

To follow a request through an actual service, read a request through the system. It connects input preparation, queues, caching, model execution, and output, then uses a chunking example to explain first-token versus streaming latency. If you already deploy models, jump to the KV memory budget or cache eviction.

Once decode and caching are clear, DFlash and MTP examines reducing target-model rounds: follow drafting and verification, then rejection, cache rollback, and costs. Beginners do not need every acceleration method first.

A small performance diagnosis

Suppose a request waits 800 ms for its first token, then receives a token every 20 ms. These intervals answer different questions. The first can include queuing, prefill, and the first generation step; the second is closer to ongoing decode, and the decode roofline example shows why, at small batch sizes, it is mostly limited by memory bandwidth. Reporting only average tokens/s hides the distinction.

If inputs grow longer, inspect prefill and cache reads. If concurrency delays the first token, inspect queuing, scheduling, and resource contention. This example illustrates measurement decomposition, not proof of a particular bottleneck.

Training-side memory belongs in pretraining and training mechanics, rather than an inference-cache checklist. Version-specific claims still need the model report and implementation.