Diffusion language models: review questions

Forward process: at time $t\in[0,1]$, each position is independently replaced by [M] with probability $t$, and once it becomes [M] it never changes back (the absorbing state). Training objective: draw $t\sim U(0,1]$ and the corresponding $x_t$, compute the cross-entropy only at masked positions, and multiply by $1/t$ (LLaDA, Eq. 3). At small $t$ only $tL$ positions contribute on average; after the $1/t$, every $t$ carries an expected weight of $L$ tokens, and the whole objective becomes an upper bound on the negative log-likelihood (NLL). With another masking schedule, the continuous-time weight is $-\alpha_t'/(1-\alpha_t)$; the objective itself does not change, only the gradient variance. The model does not need to know $t$, since the number of blanks is already in the input.

The network can be the same bidirectional Transformer; the difference lies in the distribution of masking ratios and the weighting. BERT masks a fixed ~15% and averages over selected positions; it never learns to start from all blanks and has no likelihood interpretation. Masked diffusion draws a $t$ per example, so it practises everything from almost no masking to full masking, and the $1/t$ ties it to maximum likelihood; MDLM writes it as a weighted average of MLM losses. LLaDA also notes that MaskGIT drops the $1/t$ and so loses the link to maximum likelihood.

In training, the objective averages over all generation orders; at inference, blanks can be filled in any order. With two tokens: both are masked with probability $t^2$ and exactly one with probability $t(1-t)$ each; after multiplying by $1/t$ and integrating, both terms get weight $1/2$, so the objective is the average of the left-to-right and right-to-left autoregressive (AR) losses. With the teaching numbers that is $(1.833+2.996)/2=2.414$; the negative log-likelihood of the model actually generating the sequence is $-\ln 0.105=2.254$, so it is an upper bound, loose when the two orders disagree. At any length, RADD's Theorem 2 proves equivalence with the any-order autoregressive objective. The price is that hard orders must be learned too: the same validation loss takes about 16 times the compute of autoregressive models (Nie et al., 2024-10).

The tokens in one step are each chosen from their marginal distributions without seeing one another. If the answer is full house half the time and high card the other half, committing both in one step gives a valid hand only half the time. Fast-dLLM decides by confidence: commit every position above a threshold (0.9 by default), or the single most confident one if none passes; its theorem gives a condition: if $n$ positions all have confidence above $1-\epsilon$ and $(n+1)\epsilon\le1$, taking per-position maxima agrees with taking the joint maximum. The cost of a fixed step count has been measured by a third party: with LLaDA-8B-Instruct at generation length 128 and block size 32, a fixed 4 per step drops GSM8K from 73.6 to 61.7 (73.6 being the one-per-step baseline in the same setting). Feng et al.'s Theorem 4.4 gives a linear-step lower bound for a worst-case HMM with a 16-token vocabulary, under their factorized reverse process and small learning-error assumptions. Its sequence error measures probability outside the HMM's support; it is not a universal limit on reasoning speedups. Compare latency at matched task accuracy.

In an autoregressive model, the K and V at position $i$ depend only on $x_{\le i}$, which never change. Under bidirectional attention, from layer 2 onward every position has read the blanks that change at every step, so even the prompt's K and V change, and LLaDA recomputes the whole span at every step. Fix one is approximate caching (Fast-dLLM): KV is very similar between adjacent steps, so reuse it within a block and refresh after the block, which requires checking accuracy separately. Fix two is block diffusion: left to right across blocks, diffusion within a block, so the KV of finished blocks can be cached exactly; smaller blocks are closer to autoregressive and give better perplexity, but less parallelism.

With the second note's teaching model (linear layers only, numbers from the roofline note: 7B, BF16, H100 SXM5 peak), the lower bound for one forward pass is $\max(4.18\,\text{ms},\ n\times0.01415\,\text{ms})$; below about 295 positions, the weight read dominates. For one request with a 512-token prompt and 256 generated tokens: AR takes about 1,076 ms; whole-sequence diffusion without a cache computes 768 positions per step and takes 2,782 ms at one token per step, slower than AR; blocks with $G=32$ plus a cache, at 4 per step, take about 275 ms, 3.9 times faster. As concurrency rises, diffusion computes $G/\tau=8$ positions per committed token and caps at 8,834 tok/s against AR's 70,671 tok/s, so AR catches up at about 37 concurrent requests. The low-concurrency speedup is therefore about $\tau$, and the throughput ratio at compute saturation is about $\tau/G$. Third-party measurements and Google's own notes on DiffusionGemma point the same way.

Four things: at what concurrency; is the baseline on the same GPU and the same engine; what is the quality at that speed, and with which threshold; and does the timing include prefill, the per-block commit pass and fixed overheads. Examples: Gemini Diffusion's 1479 tokens/s excludes a 0.84 s overhead; the comparison speeds in the Mercury report are Artificial Analysis's median measurements across each model's cloud providers; in the LLaDA2.0 paper the diffusion models and baselines run on different engines (dInfer versus SGLang), and when the SGLang blog puts both on one engine the ratio falls from about 2.1 to about 1.9; Mercury 2.5 does not name the GPU model. Quality has to be read at the same time: on the DiffusionGemma model card, most benchmarks are below the same-size Gemma 4. As of October 2026, all of these figures come from the vendors or the methods' authors.

DFlash uses block diffusion only to draft: a small model reads the target model's features, writes a block in one forward pass, and the autoregressive target model verifies it and keeps only the contiguously accepted prefix, so the output distribution equals the target model's and the quality risk is not on the diffusion side. A native diffusion LM decides the final tokens itself and bears the risk of inconsistent parallel commits and lower same-family quality. The speed accounting has the same shape for both: each forward pass computes a whole block, the more tokens accepted (or committed) the better the trade, and the gain shrinks at high concurrency.