DFlash: drafting, verification, and speed

Checked: 2026-10-10 · Prerequisite: generation and KV cache · Start with the flow and example; read sampling, caches, and the experiment table when implementing.

Generating several tokens can all sound like “parallel prediction.” First ask: who proposes, who decides, and what happens to the work behind a wrong guess? MTP concerns training objectives and prediction modules; DFlash uses a small drafter to accelerate a target model. The names describe different levels of a system.

Separate 3 decisions

Early MTP work uses multiple heads on a shared trunk; DeepSeek-V3 uses sequential MTP modules. Do not draw every MTP design as the same architecture. These modules can supply speculative drafts, but verification is a separate part of producing reliable output.

One DFlash generation cycle

The original DFlash paper freezes the autoregressive target and trains a lightweight block-diffusion drafter. Target context features enter each draft layer's K/V; a confirmed anchor token and masked positions produce parallel proposals in one forward pass. The target itself is not replaced by a diffusion LLM.

Training must respect the same information boundary: full answers may be available, but a predicted block must not read target features containing its future answers. The paper uses random anchors and specialized attention masking, with greater loss weight on early positions. The project explanation describes feature injection and shared embedding/LM-head components.

Why emphasize early positions? The following example needs no model run.

Why can't later correct guesses simply stay?

Let A B X D denote 4 draft tokens without assuming a tokenizer. Start with greedy decoding: the target chooses its highest-probability token.

The cycle can emit A B C, not declare A B C D correct. Replacing X with C may change the next conditional distribution. What survives is a continuous accepted prefix, not every matching position scattered across the block.

The following implements only greedy verification control flow. target_choices[position] must correspond to that draft prefix; the final element is the bonus token when all proposals match. Real inference also needs logits, masks, and caches. This is not a complete DFlash implementation.

def verify_greedy_block(draft, target_choices, eos_token=None):
    if len(target_choices) != len(draft) + 1:
        raise ValueError("expected one target choice per draft token plus a bonus")
    output = []
    accepted = 0
    for position, candidate in enumerate(draft):
        expected = target_choices[position]
        if candidate != expected:
            output.append(expected)
            return output, accepted
        output.append(candidate)
        accepted += 1
        if eos_token is not None and candidate == eos_token:
            return output, accepted
    output.append(target_choices[-1])
    return output, accepted


emitted, accepted = verify_greedy_block([10, 20, 99, 40], [10, 20, 30, 40, 50])
assert emitted == [10, 20, 30] and accepted == 2

This also explains cache rollback. Verification may compute K/V for X and its suffix; those states cannot remain in the committed prefix. Retain only valid prefix state. If the replacement C has not gone through a forward pass, its K/V still needs to be computed later. Emitted-token count is not automatically the current cache length. Draft and target caches must each stay aligned.

Sampling with temperature is not argmax matching

Greedy consistency is straightforward. Sampling must preserve the target's distribution, not identical text under the same seed. Speculative sampling uses acceptance/rejection correction rather than accepting plausible-looking tokens.

For target distribution $p$, actual proposal distribution $q$, and proposal $y\sim q$, accept with

$$\min\left(1,\frac{p(y)}{q(y)}\right).$$

On rejection, sample from normalized $[p-q]_+$. See the MTP sampling example for an exact 2-token calculation.

“Actual proposal” matters. After top-k, temperature, or a selector changes the proposal, the verifier needs the corresponding $q$, not the old softmax. Parallel guesses are not automatically samples drawn sequentially from target conditionals. The proof concerns distributions; it cannot repair invalid caches, masks, or probability alignment.

Account for a complete cycle

Let the baseline take $T_{\mathrm{AR}}$ per output token. Drafting, verification, and other cycle costs are $T_d,T_v,T_o$. Let $E[N]$ count actual new emitted tokens, including accepted proposals and the target-supplied token, but not the already-known anchor. Roughly,

$$\text{speedup}\approx \frac{T_{\mathrm{AR}}E[N]}{T_d+T_v+T_o}.$$

This is an amortized fixed-load estimate, not a queueing model. The following numbers are teaching assumptions, not DFlash benchmarks: 10 ms/token baseline, 3 ms draft, 12 ms verify, and 1 ms other cost.

Change the costs in this small calculation:

import math


def cycle_speedup(baseline_ms, draft_ms, verify_ms, overhead_ms, emitted_tokens):
    values = (baseline_ms, draft_ms, verify_ms, overhead_ms, emitted_tokens)
    if not all(math.isfinite(value) for value in values):
        raise ValueError("inputs must be finite")
    if baseline_ms <= 0 or emitted_tokens <= 0:
        raise ValueError("baseline and emitted tokens must be positive")
    if min(draft_ms, verify_ms, overhead_ms) < 0:
        raise ValueError("costs must be nonnegative")
    cycle_ms = draft_ms + verify_ms + overhead_ms
    if cycle_ms <= 0:
        raise ValueError("cycle cost must be positive")
    return baseline_ms * emitted_tokens / cycle_ms


assert cycle_speedup(10, 3, 12, 1, 4) == 2.5
assert cycle_speedup(10, 3, 12, 1, 1) == 0.625

A longer block may face more suffix rejection and higher verification/cache cost. Parallelism is not free. Under concurrency, drafter compute and memory can also affect other requests. At small batch sizes decode mostly waits on weight reads, so computing a few extra positions adds little time; that slack disappears as the batch grows, see the decode roofline example. A batch=1 latency improvement does not establish a whole-server throughput gain.

Developments in 2026: making parallel guesses coherent

Original DFlash proposes positions in parallel; a later position has not observed the token just selected at its predecessor. That leaves room to improve local coherence.

The official DFlash 2 introduction, dated 2026-08-18, keeps the parallel backbone, adds lightweight adjacent-candidate path selection, and uses short convolutions for local block mixing. Path traversal still has sequential work; parallel drafting does not mean the system has no dependencies.

An original miniature example: candidate columns contain {New, Los} and {York, Angeles}. Independent top-1 choices might produce New Angeles; adjacent-pair scoring can favor New York. This explains coherence, not access to the correct answer. The target still verifies the result.

Another route is D-Loop, submitted 2026-10-05: reuse the drafter to predict a suffix conditioned on a selected prefix. It also changes training; running original DFlash weights twice does not reproduce the reported gains. In §3.2, shared weights learn prefix prediction from an anchor-only input and suffix prediction from a revealed ground-truth prefix. Inference uses a proposed prefix instead. This mechanism has been checked, but experiments have not been reproduced or used to rank hardware.

Versions also affect usability. The current official repository lists DFlash 2 and backend-specific support. The old project page still describes vLLM integration as in progress, but as of 2026-10 both vLLM v0.31.0 ( method: dflash in the speculative config) and SGLang v0.5.21 ( --speculative-algorithm DFLASH) ship DFlash in their main code. Check the commit, target/draft checkpoints, and backend before running, rather than copying an old installation command.

What evidence should precede deployment?

Establish correctness in a controlled setting before choosing where to enable it. The code here checks local logic and arithmetic, not full drafter training or GPU serving performance. For parallel target scoring, revisit causal masks and decoding; for training-target alignment, read MTP.