Nemotron 3.5 Bolt: Heterogeneous LatentMoE and the 24 GB Budget

AI disclosure: This article was prepared with generative AI assistance. Editorial responsibility: Florian Zimmermeister (flozi00). Legal notice and contact.

NVIDIA released Nemotron 3.5 Bolt 74B-A9B 24GB on October 15, 2026, with quantized Safetensors, GGUF, BF16 and companion drafter checkpoints. The useful architecture detail is not simply “a 74B model in two bits.” Bolt combines different expert counts and widths in different blocks, a reduced-width routed-expert path, several weight formats and a hybrid attention/recurrent cache.¹²

This article audits the quantized Safetensors revision c6cc2e8ffa353403213d6b9b18aaf8eb684d705a and GGUF model-card revision e0fecb69cb3e5e4c5e8a056fa2aec98c78e91246, checked on October 17, 2026. Derived byte counts are not GPU allocations or throughput measurements. The quantization guide explains the general precision trade-off; the KV-cache guide explains why weights are only one part of a serving budget.

The saved architecture is heterogeneous, not a uniform expert stack

The language backbone contains 88 sequential blocks: 40 Mamba-2, 40 MoE and eight attention blocks. These are separate residual blocks, not 88 conventional layers each containing attention plus an FFN. The attention indices are 7, 16, 25, 36, 47, 58, 69 and 78, zero-based.²³

For example, block 3 has 240 routed experts of width 1,536, block 48 has 496 of width 2,560, and block 79 has 496 of width 2,560. The authoritative values are in block_configs. The custom configuration builds a separate base configuration for each block and removes blockwise fields from its global configuration. Reading a leftover top-level num_experts: 512 or intermediate_size: 2688 as the executed geometry would therefore be wrong.²⁴

NVIDIA attributes this layout to budget-conditioned architecture search and distillation from Super 3.5 VL, followed by recovery and quantization-aware distillation. The published search budget includes later quantization and runtime states at batch one and 128K context. This describes how the fixed checkpoint was constructed; the deployed model does not rerun architecture search or resize experts for every request.⁵¹

Follow one token through LatentMoE

For hidden vector $x\in\mathbb{R}^{4096}$, routing reads the full-width input. The routed branch separately projects it to $z\in\mathbb{R}^{1024}$. In block $l$, expert $e$ has matrices $U_{l,e}\in\mathbb{R}^{I_l\times1024}$ and $V_{l,e}\in\mathbb{R}^{1024\times I_l}$. It computes squared ReLU, not a three-matrix SwiGLU FFN.³

Squaring is elementwise. The shared expert operates on the original 4,096-wide input, with $U_{l,s}\in\mathbb{R}^{5376\times4096}$ and $V_{l,s}\in\mathbb{R}^{4096\times5376}$. It is not one of the nine selected experts and is not inside the 1,024-wide latent branch. The equation describes the MoE mixer; the surrounding block adds its residual separately.³

The router computes FP32 logits and sigmoid scores $s_e$. It selects nine experts using $s_e+b_e$, where $b_e$ is a correction bias, but constructs mixture weights from the original unbiased scores. The normalization and scaling are:³

Here n_group=1 means expert grouping does not restrict the candidate pool. Do not confuse it with n_groups=8, which controls Mamba's B/C groups. The latent width quarters each routed expert's two matrix sizes compared with a hypothetical otherwise identical 4,096-wide expert. Projection and shared-expert costs remain; this dimensional comparison is not a measured fourfold model speedup.

IQ2_XXS is a codebook format, not four independent levels per weight

The checkpoint specifies 66 bytes for 256 weights, giving $66\times8/256=2.0625$ bits per weight. The pinned GGML structure contains a two-byte FP16 block scale and 64 packed bytes. Its decoder interprets each group of 32 weights as four eight-weight grid lookups, four sign selectors and one local scale.²⁶⁷

For a 32-weight group, the four-bit selector $a\in\{0,\ldots,15\}$ gives multiplier $d(a+0.5)/4$, with FP16 block scale $d$. Each eight-weight subvector then takes eight magnitudes from a 256-entry grid and a sign pattern from the 128-entry sign table. These are coupled vector representations, not 256 unrelated two-bit scalar codes. The two-bit label describes average storage; it does not establish native INT2 arithmetic on the GPU.⁷

The matrix multiply still consumes activations at another precision. The audited vLLM block-quantized MoE path explicitly requires BF16 activations. Repacking or dequantizing tiles for a kernel does not mean the whole model must reside as a dense BF16 copy, but it does introduce kernel-specific preparation, scratch and compatibility constraints.⁸

The actual checkpoint is mixed precision

The quantization configuration lists different target modules. In particular, the routed and shared expert matrices use IQ2_XXS, while the latent projections are higher precision. Q8_0 itself occupies 34 bytes per 32 weights = 8.5 bits/weight, including its FP16 scale.²⁶

The embedding row is a concrete qualification to the broad precision description in NVIDIA's blog: the pinned Safetensors export has no embedding entry in quantized_layers, and the first shard's actual header declares language_model.backbone.embeddings.weight as BF16, with 1,073,741,824 data bytes. This audit read the shard header, not the weight payload. Do not transfer a precision claim between the Safetensors and GGUF exports without checking their tensors.²⁹

For these two-matrix experts, parameter arithmetic is straightforward. Sum over the 40 block configurations, with latent width $L=1024$, routed count $E_l$ and intermediate width $I_l$:²³

This is the logical packed payload of those expert matrices, excluding runtime layout padding and other tensors. Selecting nine experts reduces per-token work; it does not remove unselected expert weights from a resident model. The released Safetensors index reports 24,079,678,824 total tensor bytes, approximately 24.08 decimal GB or 22.426 GiB, for the whole exported model. Neither number is peak device memory: loading, prepared layouts, recurrent state, KV, vision work, drafter, workspaces and allocator overhead can require more.¹⁰

Why distillation can help, without making two bits lossless

A useful general identity explains why rounding error alone is insufficient. Let $\widehat W=W+\Delta W$ and let $\Sigma_x=\mathbb{E}[xx^{\mathsf T}]$ be the input second-moment matrix. Then the expected squared linear-output error is:

The same weight error has different consequences along differently excited input directions. Across nonlinear layers, routing and recurrence, local output errors can change later computation too. Quantization-aware distillation exposes the student to quantization effects while training it to recover a teacher's behavior; it can adapt weights rather than merely round a frozen network. NVIDIA reports using QAD for Bolt, but this identity is an explanatory derivation, not a specification of its training loss or proof of preserved accuracy.⁵

A larger lower-bit model can retain useful capacity under a fixed memory budget, but that is a workload-dependent empirical trade-off. It does not imply every 74B two-bit checkpoint beats every smaller four-bit model. No independent quality or latency comparison was performed for this article.

The hybrid runtime state still needs a budget

Each of the eight attention blocks caches separate K and V with logical shapes $[B,2,T,128]$. Their combined BF16 payload is 8 KiB per token per sequence: 1 GiB at 131,072 tokens and 2 GiB at 262,144. Raw FP8 elements halve those payloads before scales, paging and engine-specific overhead. GGUF q8_0 is not identical to one-byte FP8 storage.²³

Each Mamba block instead keeps an SSM state $[B,128,64,128]$ and a convolution state $[B,10240,4]$, where $10240=128\times64+2\times8\times128$. For batch one across 40 blocks, one BF16 state copy totals 80 MiB of SSM state plus 3.125 MiB of convolution state. This is independent of sequence length, but not of concurrency or the number of snapshots retained by the engine.³

The saved mamba_ssm_cache_dtype field says float32; the published vLLM recipe explicitly requests BF16 SSM and convolution caches. At FP32, the SSM part alone would be 160 MiB. Inspect the executed dtype, not just the weight dtype or one config field. These floors exclude rollback/alignment state for speculation, prefix-cache copies, padding and allocator reservations.²⁵

Released serving paths have different boundaries

The pinned GGUF card distinguishes a 28GB recipe with 65,536 context, q8_0 KV and up to six draft tokens from a 32GB recipe with 262,144 context and up to seven draft tokens. Its hardware recommendations are respectively RTX Spark 64GB or discrete GPUs with 32GB+, and RTX Spark 128GB or discrete GPUs with 48GB+. Its shorter quick-start snippet instead requests 98,304 context. These are different recipes, not evidence of universal 24GB full-context residency.¹¹

The vLLM recipe names image nightly-77154ca6116ae34cbfe314beef1016bde9b63989, requests b12x linear/MoE backends, FP8 KV and BF16 recurrent caches. At that source revision, native block-quantized expert loading rejects non-SM12x devices, non-BF16 activations and expert-parallel/EPLB layouts; it supports tensor parallelism without EP/EPLB. Broad A100/H100 wording in the model-card summary is not enough to establish support for this particular packed path.⁵⁸

The standalone b12x repository is now archived; its pinned README points development to flashinfer/experimental/b12x and identifies the migration commit. Historical pip install b12x instructions and newer FlashInfer packaging must not be mixed without an integration test. The kernel subsystem targets SM120/SM121, and individual operations may have narrower limits. This is a version-specific deployment boundary, not a recommendation to replace a working serving stack blindly.¹²

Bolt's backbone has no built-in MTP layer. The released LiLiCorr drafter is a separate model used through DFlash; it adds memory and must be loaded alongside the target. The GGUF card also loads a separate multimodal projector. Treat the target, projector, drafter, caches and workspaces as separate budget items. Pin their revisions and the tested runtime build before deployment.²¹¹

The practical conclusion is a budget, not a slogan: Bolt saves weight bytes through heterogeneous latent experts and codebook quantization, while retaining higher precision elsewhere. Its hybrid backbone limits which blocks accumulate token-wise KV, but long context and speculative decoding still have costs. The release is usable software with downloadable weights; the appropriate context, concurrency and hardware remain matters for workload-specific validation. We did not execute the model or benchmark its GPU kernels.

Primary sources