Nemotron 3.5 Bolt E4B: Block-Parallel Speculation with DFlash

AI disclosure: This article was prepared with generative AI assistance. Editorial responsibility: Florian Zimmermeister (flozi00). Legal notice and contact.

NVIDIA released Nemotron 3.5 Bolt E4B BF16 and its target-specific DFlash NVFP4 drafter on October 23, 2026. This is a usable serving pair rather than a paper-only prototype: NVIDIA publishes both checkpoints and a vLLM 0.31.0 command. The target has 8B total and 4B effective parameters; the companion is a 1.03B-parameter, five-layer Qwen3-style draft network.¹²

Despite the shared “Bolt 3.5” name, this is not the 74B-A9B hybrid model covered in the LatentMoE and IQ2 article. E4B is a 42-layer Gemma 4 text backbone. Its DFlash companion is also different from the LiLiCorr drafter released for 74B Bolt. A drafter is fitted to one target's vocabulary, hidden width and selected internal layers; it is not a generic small language model.

This article audits target revision 49843eed8db1f1c616c47af75cfd36e07aec0667, draft revision 913d13c49fbbb2b477dce78552a31ab650d5b693, vLLM db9527a46873454610df6dbedf79a36d6bf1a7f6, and NeMo AutoModel 6b0f669ec07e4b9d0f37c92162384b3c7cc97c15, checked on October 24, 2026. Byte counts below are logical payloads, not measured GPU allocations. We did not execute the checkpoints or benchmark a GPU.

The released serving cycle

Ordinary autoregressive decoding runs the full target once for every new token. DFlash instead amortizes one target verification over a block:

1. the target supplies the last verified token and hidden states captured after zero-based layer IDs 5, 11, 17, 23 and 35;
2. the drafter forms an eight-position block: one real anchor followed by seven mask tokens;
3. one non-causal five-layer forward predicts all seven future positions in parallel;
4. the target evaluates the proposed continuation in one pass; and
5. exact speculative verification accepts the longest valid prefix and samples a correction when required.

The output still comes from the target distribution. “Non-causal” applies inside the draft block, not to target verification: a proposal is cheap, but it cannot authorize itself. The released block size is eight, so the maximum acceptance length is eight output tokens per cycle, including the target's correction or next token.²³⁴

Five target states become persistent draft context

Let the five selected target tensors be $H^{(j)}\in\mathbb{R}^{B\times S\times2560}$. The runtime concatenates them on the feature axis and projects them back to the draft width:

The resulting context is projected into K/V entries and inserted into every draft layer's cache. For the eight block positions, each layer forms Q with shape $[B,32,8,128]$ and K/V with shape $[B,8,8,128]$ before adding the cached context. Four query heads share each KV head. Per-head Q/K RMS normalization is applied before RoPE. The released vLLM implementation can fuse unquantized K/V projections into one matrix multiplication; packed quantized weights instead go through the configured quantization method per layer.⁵⁶

The chosen target indices are not arbitrary positions in a uniform stack. E4B has 35 sliding-window layers and seven full-attention layers, with a full layer every sixth layer. The drafter reads five of those full-attention outputs: 5, 11, 17, 23 and 35. It does not capture full-attention layers 29 and 41. This gives the small draft network compressed global features while its own attention remains limited to the latest 1,024 tokens.⁷⁵

During training, the anchor is real and later positions are masked. Positions inside one block may communicate bidirectionally, but the mask prevents leakage from later blocks. NVIDIA weights earlier prediction positions more heavily, using $w_k=\exp(-(k-1)/\gamma)$ in the published method. These are training mechanics; the release fixes the block to seven proposals at inference.³⁴

Why exact verification preserves the target

For one position, let $q(x)$ be the drafter's probability and $p(x)$ the target probability. If the draft samples token $x$, speculative sampling accepts it with

If it rejects, it samples from the normalized positive residual

The probability mass accepted through $q$ is $q(x)a(x)=\min(p(x),q(x))$. The rejection branch contributes the missing positive part $[p(x)-q(x)]_+$. Together they reconstruct $p(x)$ after normalization. Applied sequentially across the proposed block, this accepts the longest valid prefix without changing the target distribution. For greedy decoding the rule is simpler: accept proposals while they equal the target argmax, then take the target token at the first mismatch.³⁴

This is why a weaker draft can still be useful: it does not need to be trusted for final quality. It only needs enough conditional agreement with the target to amortize verification. A better draft increases the expected accepted prefix, but an overly large draft raises the cost paid on every cycle.

The parameter count exposes the real cost

The model card reports 1,031,568,640 logical parameters. The released dimensions reproduce that number exactly:

The output head alone is 65.1% of the parameters. It is an NVFP4 copy of the target head because the drafter must score the same 262,144-token vocabulary. Conversely, the token embedding is shared with the target and absent from the draft checkpoint. At BF16, avoiding a duplicate $262144\times2560$ embedding avoids a 1.25 GiB logical payload.²⁵

At an ideal four bits per parameter, all draft codes would occupy 491.9 MiB; the output-head codes alone would occupy 320 MiB. The published repository is 580 MB because real storage also contains quantization scales, BF16 exclusions, tensor metadata and alignment. “NVFP4” is therefore a compute/storage format, not a promise that the file is exactly half a byte times the parameter count.

The default revision is W4A16: NVFP4 weights with 16-bit activations. The w4a4 revision also uses NVFP4 activations and requires Blackwell activation kernels; NVIDIA specifies a CUDA 13 vLLM build on GB300. The config leaves draft KV-cache quantization disabled and excludes norms from the packed path.²⁵⁸

Cache and latency have separate break-even points

For five draft layers, eight KV heads, a 1,024-token window, head width 128 and BF16 K/V, the logical cache payload per sequence is

At 64 sequences that payload alone is 1.25 GiB, before paging, alignment and allocator reservations. DFlash also makes the target expose five hidden states and maintains proposal/rollback state. The drafter checkpoint's small file size is not the full serving overhead.

Let $C_t$ be the time for one ordinary target decode step, $C_d$ the block-draft time, $C_v$ the target verification time, and $\tau$ the measured mean output tokens accepted per cycle. Then

An acceptance length of four is therefore not a fourfold speedup. Verification processes a block efficiently, but it is not free; the draft pass, state projection, cache traffic and rejected proposals all remain. The break-even point changes with prompt length, sampling entropy, batch size and kernels.³

NVIDIA's vendor measurements use W4A16, thinking enabled, 4,096 output tokens, concurrency 16 and one GB300. Under those conditions the reported mean acceptance length is 4.01 for greedy MT-Bench and 3.65 with temperature 1; SPEED-Bench reports 3.88 and 3.50. These are acceptance measurements, not end-to-end speedups, and they must not be transferred to other hardware or traffic patterns. NVIDIA also reports roughly 1% lower acceptance for W4A16 and 2% for W4A4 versus an unquantized BF16 draft.²

Production boundary: latency path, not universal throughput path

The official DFlash example sets max-num-seqs=1. The target card's separate maximum-throughput recipe sets 64 sequences and explicitly says that no speculative decoding is best for that configuration. This is the practical dividing line: DFlash can reduce low-concurrency inter-token latency when spare compute can turn a verification pass into several tokens; continuous batching may already keep the target efficient enough that the extra draft work loses.¹

Other boundaries matter:

• Target pairing is exact. The 2,560-wide projection, vocabulary, layer indices and target head tie this checkpoint to Nemotron 3.5 Bolt E4B.
• The joint context contract is 64K text. The draft config declares 131,072 positions and its card lists multimodal inputs, but the released E4B service card specifies text/text and 65,536 tokens, and its vLLM command disables image, video and audio. Use the target's narrower deployed contract unless a separate stack is validated.
• Hardware paths differ. W4A16 is the default. W4A4 is a Blackwell-specific activation-kernel path, not a portable synonym.
• Quality is target quality, not draft quality. Exact verification preserves sampling semantics; it does not fix inaccuracies, bias or unsafe target output.
• Pin both artifacts and the runtime. Changing either target weights or hidden-state conventions can reduce acceptance or make the checkpoint incompatible.

The engineering value of this release is the separation of responsibilities. The 42-layer target remains the authority; five selected global target states condition a much smaller non-causal network; exact verification protects the distribution; and NVFP4 reduces the always-resident draft weight cost. Whether that trade is positive is measurable only at the intended context, sampling policy, concurrency and GPU.

Primary sources