Nemotron 3.5 Super VL: Where Visual Token Compression Happens

AI disclosure: This article was prepared with generative AI assistance. Editorial responsibility: Florian Zimmermeister (flozi00). Legal notice and contact.

NVIDIA released Nemotron 3.5 Super VL 120B-A12B on October 15, 2026, with BF16 and NVFP4 checkpoints and vLLM/SGLang serving recipes. The BF16 card lists OpenMDW-1.1 as the governing model license. This is a usable text/image/video-to-text model; the generic class name containing “Omni” does not establish audio support, and this checkpoint has sound_config: null.¹²

This guide audits BF16 checkpoint revision 1f8953f7f0935358e3605150d22bc15df062061c and vLLM revision fce64bc5c2c939a4521bfb9b462c8c74c1c13893, checked on October 20, 2026. It follows the visual input path, rather than repeating the Bolt LatentMoE and weight-storage guide. Counts below are source-derived, not measured latency, accuracy or peak VRAM.

What the released configuration actually connects

The language backbone has 88 separate blocks: 40 Mamba-2, 40 MoE and eight attention blocks, with hidden width 4,096. They are not 88 conventional attention-plus-FFN layers. MoE blocks select 22 of 512 routed experts and also have a shared expert; the routed latent width is 1,024. Those counts are not Bolt's heterogeneous expert geometry.²

The model card identifies the encoder as C-RADIO. Its code supplies dense patch features to the connector, not a single pooled image vector. Class and register prefixes participate in vision processing but are stripped from the feature sequence passed onward.¹³

Images: dynamic resolution before language tokens

The exported image processor targets one dynamically sized tile per image. It uses a 16-pixel patch grid, patch-budget settings of 1,024 to 13,312, and even grid dimensions for the later spatial merge. Rounding and aspect ratio determine the actual grid; these settings are not a promise that every image produces exactly the maximum token count. force_image_size: 512 is not evidence that this processor always resizes to a 512-pixel square.⁴⁵

The executable preprocessing path uses antialiased bicubic tensor interpolation with align_corners=False, followed by channelwise normalization $(x/255-\mu)/\sigma$. That detail is more precise than a “bicubic resize” label or the file's older PIL-oriented comment. Kernel, antialiasing and normalization changes alter inputs before any learned computation.⁵

For a processed image of size $H'\times W'$, define $h=H'/16$, $w=W'/16$ and $P=hw$. The following dataflow excludes the ten vision prefix tokens once stripped. The example starts with an already processed 512 × 512 image; it is not a universal resize rule.

The final vision LayerNorm before rearrangement is built here because the saved backbone has MTP enabled. It is separate from the connector's RMSNorm after rearrangement, with epsilon $10^{-5}$. Their order and axes cannot be interchanged without changing the function.⁶⁷

Pixel shuffle is a permutation; the connector is not

Despite its name, pixel_shuffle here moves four neighboring feature vectors into one wider vector. For the saved v2 layout, write $V$ for the normalized vision grid and $z$ for its rearrangement:

The element count is unchanged: $P\cdot1280=(P/4)\cdot5120$. On a divisible grid the rearrangement itself is invertible. It is not averaging and does not by itself discard three quarters of the scalar features. The learned connector then computes:

Squaring is elementwise; the two linear layers have no bias. This is a learned nonlinear projection to language width, not a lossless image codec. Relative to sending every patch through a 4,096-wide connector, the language model receives four times fewer visual positions. That is a token-count comparison, not a measured fourfold end-to-end speedup.⁶

The wrapper embeds the text prompt and replaces image-placeholder rows with these vectors. The language backbone therefore consumes visual and text positions in one sequence; it does not cross-attend to a separate permanent image memory. The LLM foundations guide describes the subsequent token-to-output loop.

Video: a learned two-frame patch, not frame averaging

For $F$ sampled frames, the temporal group count is $G=\lceil F/2\rceil$. The exported model repeats the last frame when $F$ is odd, reshapes each pair to six channels, and uses a distinct video patch projection. Each spatial patch now has $2\cdot3\cdot16^2=1536$ input scalars, mapped to 1,280 features before the vision Transformer.⁶³

This projection can learn different weights for the two frames; it is not constrained to their average. It is also not generally invertible: the linear map has rank at most 1,280 for 1,536 input dimensions. The same spatial rearrangement and connector then operate on each group's features. For an even number of frames, pairing plus spatial merging yields eight times fewer language positions than treating every raw spatial patch of every sampled frame as a separate position. The image and video patch projectors are different learned maps, so that ratio does not establish equal information retention.

The exported processor labels temporal groups with frame/timestamp text. These labels, image boundary markers, the question and the response consume additional context positions beyond the visual count. Frame sampling occurs before this path: an event missing from the sampled frames cannot be recovered by a larger language context.⁸

EVS: preserve change, not guaranteed relevance

Efficient Video Sampling (EVS) runs after vision encoding and projection. For each spatial position $s$, it compares adjacent temporal-group embeddings $e_{g,s}$ by cosine similarity. The first group receives an artificial score of 255; later scores lie in $[0,2]$ for finite nonzero vectors. For $S=hw/4$ positions per group and pruning rate $q$, the retained count is:⁹

The implementation selects the largest scores globally across the clip, with a stable ordering for ties. All first-group positions survive, but later groups need not retain equal quotas. A 70% pruning setting therefore does not remove exactly 70% from each group, and the one-group floor dominates short clips or very large $q$.

Cosine distance is a change heuristic, not a relevance estimator. Positive rescalings of the same vector have distance zero; camera motion can create large feature changes without adding useful evidence. A brief relevant detail between sampled frames is absent altogether. There is no reconstruction of deleted vectors and no guarantee that an arbitrary OCR or tracking task survives a given rate. Those are mechanism-based limitations, not measured failures of this model.

A reproducible token budget, not a throughput claim

Consider 256 already sampled frames. Under the exported aspect-preserving sizing rule with a target of 1,024 patches, 16:9 input becomes 672 × 384 pixels, a 42 × 24 patch grid. Spatial merging gives $S=21\cdot12=252$; pairing gives $G=128$. The following numbers assume exactly this resolved geometry and EVS, not every runtime's defaults.⁵¹⁰

The eight backbone attention blocks use two KV heads of width 128. Their unsharded logical KV contribution from $K$ retained visual positions is $M=2\cdot8\cdot2\cdot128\cdot b\cdot K$, where $b$ is bytes per scalar. That is 8 KiB per position in BF16, excluding all other positions and states.²

These counts exclude timestamp/marker/text tokens, Mamba states, MTP state, vision activations, embedding caches, weights, parallel replication and allocation overhead. EVS reduces the downstream sequence and its attention cache; it does not skip the vision encoding of subsequently deleted features. The pinned vLLM encoder uses frame microbatches of at most 128, aligned to temporal groups, before pruning. A smaller final token count is not proof of a proportionally smaller vision peak or latency.⁷

The technical report's Table 12 keeps the vision encoder and connector in BF16 even in its released aggressive NVFP4 recipe. “NVFP4 model” must not be read as every tensor or operation being four-bit. This statement is the report's precision policy, not a new audit of every quantized shard.¹¹

Runtime boundaries worth checking before deployment

Pruning defaults differ. The checkpoint stores video_pruning_rate: 0.7. The pinned native vLLM model instead reads multimodal_config.video_pruning_rate; that runtime field defaults to None, and the model's EVS branch requires a positive value. Set and record the resolved runtime option, such as --video-pruning-rate 0.5, rather than assuming the checkpoint value is active. vLLM also uses its own processor; verify resolved video sizing and aspect-ratio settings, not just the exported HF preprocessor JSON.⁷¹²¹⁰

Rectangular video requires special attention in the exported HF path. Its pruning call infers both spatial dimensions from int(sqrt(S)). With $S=252$, that gives $15\times15=225$, which cannot reshape the 252 positions. The source-level shape mismatch can be reproduced without weights. In contrast, the pinned vLLM native path computes rows and columns from actual frame dimensions. This is a limitation of that exported pruning path, not evidence that native vLLM cannot process rectangular video.⁶⁷

Capacity and batching are separate contracts. The backbone configuration allows 1,048,576 positions; the exported image-preprocessor JSON carries max_model_len: 16384 for its sizing calculation. Neither automatically reserves space for every image, the complete text prompt and the answer. The exported processor also asserts a single text item for video. Validate the actual processor output, placeholder count, combined sequence budget and serving batch behavior rather than generalizing that reference assertion to all native engines.⁴⁸

MTP support is runtime-specific. The checkpoint index lists language_model.mtp.* tensors, but the exported HF wrapper declares an unexpected-key ignore pattern for mtp.* and does not build the draft module. The model's MTP head is recursively shared; native vLLM/SGLang recipes enable it. A plain wrapper load must not be advertised as executing that same speculative path. No speculative speedup or acceptance rate is measured here.¹³⁶¹

Before production, pin the model and runtime revisions and test resize parity, odd-frame padding, rectangular grids, actual token counts, EVS-sensitive accuracy and memory peaks at the intended concurrency. Media-loading permissions should be restricted to required inputs; a permissive example command is not a security policy. This article performs a code/configuration audit and CPU shape/count checks, not GPU inference or a quality benchmark.

For the rest of the memory budget, continue with the KV-cache guide. For a different visual fusion boundary, Cloudflare Clef Omni ends in option scoring rather than this model's autoregressive text generation.

Primary sources