Diffusion language models: decoding steps, caching and real speed

LLaDA 8B Instruct's GSM8K score was measured with one token committed per forward pass: a 512-token answer took 512 steps (LLaDA, Appendix B.6). When a third party, with generation length 128 and block size 32, made the same model commit a fixed 4 tokens per step, GSM8K dropped from 73.6 to 61.7 (Kim et al., 2025-10). How much faster a diffusion model can be depends on two numbers: how many tokens it can safely commit per step on average (call it $\tau$), and how many positions each step recomputes (call it $G$). At low concurrency, decode is held back by memory bandwidth while compute sits idle, so speed can rise by up to about $\tau$ times; once concurrency is high enough to use all the compute, each token costs about $G/\tau$ times more computation, and the advantage disappears or even reverses.

Draft, pending review. The site owner has not reviewed this note yet; facts, emphasis and wording may change. Check the primary sources linked in the text before relying on it.

This note follows the masked diffusion objective and does its accounting with the notation and H100 numbers from the roofline note.

What one step does: predict every blank, keep the confident ones

LLaDA's sampler (§2.4) starts from an answer that is entirely blank; generation length and number of steps are both hyperparameters. One step from time $t$ to $s$: feed the prompt and the partially masked answer to the model, predict every blank in one forward pass, then mask about $s/t$ of those predictions again. The default "low-confidence remasking" keeps the most confident predictions and masks the rest back; anything after EOS is discarded at the end.

Walk through the tea is cold: generation length 4, two steps, two positions kept per step. The confidences are teaching numbers.

In step 1 tea has only 0.3 and is masked again; in step 2, with the and is as context, it rises to 0.7. "Remasking" here applies only to this step's new predictions: the committed the and is never change afterwards. As pseudocode:

def sample(model, prompt, gen_len, steps):
    x = prompt + [MASK] * gen_len
    for k in range(steps):
        probs = model(x)                          # one full bidirectional forward pass
        masked = [i for i in range(len(x)) if x[i] == MASK]
        n_keep = len(masked) // (steps - k)       # spread remaining blanks over remaining steps
        conf = {i: probs[i].max() for i in masked}
        for i in sorted(masked, key=conf.get, reverse=True)[:n_keep]:
            x[i] = probs[i].argmax()              # commit; never revisited
    return x

Can committed tokens be changed? By default, masked diffusion cannot, and that separates it from the methods below. Mask-Predict (Ghazvininejad et al., 2019-04, EMNLP 2019), for machine translation, repeatedly masks and re-predicts the least confident words among all positions. ReMDM (Wang et al., 2025-03, NeurIPS 2025) adds a sampler that remasks already decoded tokens of a pretrained masked diffusion model; more steps bring quality closer to autoregressive (AR) models. LLaDA2.1 (2026-02) adds "change a word into another word" editing on top of "fill a blank with a word". DiffusionGemma's model card says the tokens not selected at each step are fully renoised, and its developer guide says a token whose confidence drops can be renoised and replaced.

How many per step: why parallel decoding writes "high house"

The tokens committed together in one step are each chosen from their own marginal distribution, without seeing one another. Fast-dLLM (Wu et al., 2025-05) illustrates this with poker hands. Suppose the answer is full house half the time and high card half the time (a teaching distribution). Position 1's marginal is full 0.5, high 0.5; position 2's is house 0.5, card 0.5. Sampling both positions in one step produces a valid hand with probability only $0.5\times0.5+0.5\times0.5=0.5$; the other half is full card or high house. In two steps, committing one first fixes the second, and validity returns to 1. The hot tea example from the previous note behaves the same way: committing both tokens in one step generates it with probability $0.2\times0.1=0.02$, against 0.105 when the blanks are filled one at a time.

Fast-dLLM decides how many to commit by confidence: every position whose confidence exceeds a threshold (0.9 by default) is committed, and if none does, only the most confident one. Its Theorem 1 says: if $n$ positions all have marginal confidence above $1-\epsilon$ and $(n+1)\epsilon\le1$, then taking each position's maximum and taking the maximum of the joint distribution select the same sequence. With three positions all above 0.95, $(3+1)\times0.05=0.2\le1$, so they can be committed together; in the poker example $\epsilon=0.5$ and $n=2$, $3\times0.5=1.5>1$, so there is no guarantee, and the threshold rule commits only one first. As a result $\tau$ changes with the content: spans that are easy to pin down can be committed many tokens per step, hard places one at a time; DiffusionGemma's model card also says that simple prompts and structured tasks such as code need fewer denoising steps.

The cost of a fixed $\tau$ has been measured by a third party. Kim et al. (2025-10) used LLaDA-8B-Instruct with generation length 128, block size 32 and Fast-dLLM's approximate cache switched on:

The 73.6 is the one-token-per-step baseline in this setting. It is not the same setting as LLaDA's self-reported 69.4 (generation length 512, no blocks), so the two cannot be compared directly; what the table compares is the drop from left to right.

Adapting by confidence does much better, but those numbers come from the method's own authors: Fast-dLLM reports LLaDA on GSM8K (5-shot, generation length 256, A100 80GB) going from 79.3 to 78.5 while throughput rises from 6.7 to 54.4 tokens/s, 8.1 times, from caching and parallel decoding combined. Google's model card self-reports that DiffusionGemma generates 15–20 tokens per forward pass, with at most 48 denoising steps.

Feng et al. (2025-02), Theorems 4.2–4.4 study a narrower question: how the step count depends on the target distribution and the error metric. For an n-gram language, Theorem 4.2 gives a step count independent of sequence length, at fixed n-gram order and accuracy, under its learning-error and length assumptions. Theorem 4.4 constructs a worst-case hidden Markov model (HMM) with a 16-token vocabulary. For the factorized masked reverse process studied in the paper, under Assumption 4.1 with per-token reverse-distribution KL error $\epsilon_{\rm learning}<1/128$, achieving low sequence error requires at least linearly many steps. Here sequence error means generating a sequence outside the HMM's support, not a measured error rate on arbitrary reasoning tasks.

The result explains why a good perplexity score need not imply reliable joint choices. It does not rule out speedups on every reasoning distribution or for every sampler. For a real model, compare latency at the same task accuracy; neither a worst-case bound nor a tokens/s figure settles that comparison alone.

KV cache: why bidirectional attention cannot be cached

In an AR model, the K and V at position $i$ depend only on $x_{\le i}$, which never change again, so the cache is exact (see the KV cache note and why a cache must be equivalent). Under bidirectional attention, from layer 2 onward every position's hidden state has read every position, including the blanks that change at every step, so even the prompt's K and V change. LLaDA §2.2 states outright that it is incompatible with KV caching; every step recomputes the prompt and the entire answer.

There are two fixes:

• Approximate caching. Fast-dLLM observes that the KV at adjacent steps within a block is highly similar, so within a block it reuses the KV of the prefix (PrefixCache) or also the suffix (DualCache), refreshing everything after each block. It is an approximation; the authors report accuracy within 1–2 points of the baseline.
• Change the structure: block diffusion. Block Diffusion (BD3-LM) runs left to right across blocks and diffusion within a block; earlier blocks never attend to later ones, so the KV of finished blocks can be cached exactly. Block size 1 falls back to AR decoding. On OpenWebText, the smaller the block, the closer perplexity gets to AR: AR 17.54; block sizes 4, 8 and 16 give ≤20.73, ≤21.68 and ≤22.27; whole-sequence MDLM ≤22.98. LLaDA2.0 is evaluated with block size 32; DiffusionGemma denoises one 256-token block at a time, and each finished block is written to the KV cache with one causal-attention forward pass (developer guide). SGLang's blog post of 2025-12-19 says it implements block diffusion on its existing chunked-prefill path.

Blocks also happen to improve quality: Kim et al. measured GSM8K at 63.2 without blocks, 73.6 with block size 32, and 75.7 with 16 (generation length 128). The cost is that blocks are serial again, and the smaller the block, the more it resembles AR decoding.

The accounting by hand (a teaching model): time on the same GPU

This section is a teaching model, not a measurement. It uses only three numbers from the roofline note: a 7B model's BF16 weights take about $1.4\times10^{10}$ bytes, the H100 SXM5 has 3.352 TB/s of bandwidth, and its BF16 dense peak is 989.4 TFLOPS. Reading the weights once takes at least $1.4\times10^{10}/3.352\times10^{12}=4.18$ ms; the compute for one position is about $2N=1.4\times10^{10}$ FLOPs, or 0.01415 ms at peak. Ignoring attention compute, KV reads and all overheads, the lower bound for one forward pass over $n$ positions is

$$ T_{\text{step}}(n)=\max\big(4.18\ \text{ms},\;n\times0.01415\ \text{ms}\big) $$

Below about 295 positions, reading the weights dominates and extra positions are nearly free; above 295, compute dominates. Below is a teaching estimate for one request with a 512-token prompt and 256 generated tokens:

Without a cache, every step recomputes 768 positions, past the 295 mark, so a single step takes 2.6 times as long as an AR step; at one token per step, the total time is also 2.6 times AR's. With blocks and a cache, each step computes only 32 positions, still in the shadow of the weight read, so committing 4 per step is almost exactly 4 times faster. If each finished block gets one extra forward pass to write the cache (as DiffusionGemma does), 8 blocks add $8\times4.18=33$ ms, for 308 ms in total, about 3.5 times faster.

Now many concurrent requests (still the teaching model, decode phase only, block diffusion with $G=32$, $\tau=4$):

Diffusion uses up the compute at about 9 concurrent requests ($9\times32=288$ positions) and caps at 8,834 tok/s from then on, exactly 1/8 of AR's cap of 70,671: each committed token costs $G/\tau=8$ positions of compute. AR catches up at about 37 concurrent requests and is faster beyond that. Both tables can be recomputed with a few lines:

W = 1.4e10 / 3.352e12 * 1e3      # reading the weights once: 4.18 ms
C = 1.4e10 / 989.4e12 * 1e3      # compute per position: 0.01415 ms
step = lambda n: max(W, n * C)   # lower bound for one forward pass over n positions
prefill = step(512)              # 7.24 ms
print(prefill + 256 * step(1), 256 * step(768), 64 * step(768))           # 1076 2782 696
print(prefill + 64 * step(32), prefill + 72 * step(32))                   # 275 308 (the latter with 8 cache writes)
for b in (1, 8, 64, 295):
    print(b, round(b / step(b) * 1e3), round(4 * b / step(32 * b) * 1e3))  # AR and diffusion tok/s at concurrency b

Putting the two regimes together:

$$ \frac{T_{\mathrm{AR}}}{T_{\mathrm{diff}}}\Big|_{\text{low batch}}\approx\tau,\qquad \frac{\Theta_{\mathrm{diff}}}{\Theta_{\mathrm{AR}}}\Big|_{\text{compute-bound}}\approx\frac{\tau}{G} $$

$\Theta$ is throughput. This accounting ignores attention and the KV cache. Once the roofline note's second table includes KV, AR's bound at context 4096 and batch 64 falls to 4,440 tok/s, so the real crossover has to be measured separately. Qualitatively, diffusion also spreads the KV read over $\tau$ tokens (one read per step, $\tau$ commits), while its attention compute also costs $G/\tau$ times more.

Measurements point the same way. Kim et al. (A100 80GB, prompt and generation both 128, block size 32) found that LLaMA-3-8B-Instruct's throughput keeps growing with batch size up to 16, while LLaDA stops improving at around 8; with a 2k prompt, batch size 1 and no blocks, LLaDA's latency is higher even for short generations. Fast-dLLM's appendix (A100, prefill 256, batch 1–32) found that at small batch sizes LLaDA with a prefix cache matches or exceeds LLaMA's throughput but falls behind as the batch grows, because LLaMA moves from bandwidth-bound to compute-bound. Google's launch post for DiffusionGemma (2026-06-10) says the same: the advantage is largest on a single accelerator at low to medium batch sizes, returns diminish in high-QPS cloud serving and serving costs may rise, and unified-memory devices such as Apple Silicon may not see the same acceleration.

How to read vendor numbers

All of the following are self-reported by the companies or the method's authors, under different conditions, and cannot be compared with each other:

A few concrete traps. In the Mercury report, the comparison models' speeds are Artificial Analysis's median measurements across each model's cloud providers, so hardware, engines and model sizes all differ; the workload is coding prompts of about 1,000 input and 1,000 output tokens. The LLaDA2.0 paper does not say which of 256 and 237 belongs to Ling-flash-2.0 and which to Qwen3-30B-A3B, and the engines differ; the SGLang blog puts both sides on SGLang (H20, TP8, small batches), where the same CAP model reaches 500 TPS against 258 and 237, about 1.9 times. LLaDA's own authors report in Appendix B.7 that, on an A100 at batch 1 with 256 output tokens, the base model reaches comparable accuracy on GSM8K and MATH at 1.5 and 1.8 times the throughput of LLaMA3 8B with a KV cache. Quality has to be read alongside: in Gemini Diffusion's own table, GPQA Diamond is 40.4% against 56.5% for Gemini 2.0 Flash-Lite, and HumanEval 89.6% against 90.2%.

When you read "N times faster", ask four things first: at what concurrency; is the baseline on the same GPU and the same engine; what is the quality at that speed, with which threshold; and does the time include prefill, the per-block commit pass and fixed overheads.

The relation to DFlash: diffusion only writes the draft

DFlash (2026-02, ICML 2026) takes a different route: a small block-diffusion model reads the target model's features and writes a block of draft tokens in one forward pass; the AR target model verifies it and keeps only the contiguously accepted prefix. The target model's output distribution is unchanged, so the quality risk does not sit on the diffusion side. The authors self-report lossless speedups above 6 times across models and tasks, and up to 2.5 times the speedup of EAGLE-3. Its accounting has the same shape as above: drafting and verification both compute a whole block, and the more is accepted, the better the trade.

What to check when you try it

Next: review questions.