Nemotron 3.5 Transcribe: From Recorded Speech to Dataset Audits

AI disclosure: Text and explanatory diagrams were prepared with generative AI assistance. Editorial responsibility: Florian Zimmermeister (flozi00). AI transparency.

A speech model is useful for more than producing another transcript. It can also provide a second opinion on an existing annotation: is speech missing from the label, is a number wrong, or is the apparent error merely a difference between spoken and written form?

NVIDIA's Nemotron 3.5 Transcribe, released on October 28, 2026, makes that workflow interesting because it combines instruction-following transcription, speaker attribution, translation and audio questions. This article explains the released architecture and proposes a dataset-auditing workflow around it. The auditing workflow is an engineering proposal, not a published quality-classification benchmark or a claim that the checkpoint has a calibrated pair-quality head.¹

The NVIDIA launch correspondence and pre-brief PDFs were used to locate and cross-check the release material. The sources linked below are public; private messages, contacts and reviewer assets are not reproduced. No model inference, corpus audit or throughput benchmark was run for this article. Source snapshot: model card and exported configurations at revision c421d6a, the October 28 technical report, NeMo Speech 7b05f096d54db47078fd1569a54d1f3ba4a4c51f, and NeMo Curator fdee8f8884cd0104772183c7938c56dbaf30841c. The architecture calculations below are derived from these sources, not measured GPU allocations.

First, identify the right Nemotron

The checkpoint is nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B. It is an offline audio-to-text model, not the separate 0.6B Nemotron 3.5 ASR streaming model and not a speech synthesizer. Streaming generated text from a completed recording does not turn it into causal microphone streaming.

Its published task boundaries matter:

German is supported for transcription, English translation and speaker labels. The task-by-language matrix, not the headline language count, should drive deployment decisions. The release provides BF16, FP8 and NVFP4 options and OpenMDW-1.1 terms; choose the actual variant and compatible runtime rather than treating the family name as a memory budget.¹

How audio and speakers reach the language model

This is a speech-augmented decoder-only language model, not a conventional ASR decoder with a text LLM appended to its transcript. The language model receives continuous acoustic embeddings in its input sequence; a preliminary transcript is not the information bottleneck. The Parallel Expert Encoder has two perception branches: a roughly 600M-parameter ASR Transformer and the roughly 99M-parameter Nemotron 3 Diarization/Sortformer speaker model. These two experts are distinct from the language model's routed MoE experts.²

The exported frontend decodes a mono waveform at 16 kHz, then computes 128 log-mel features with a 25 ms Hann window, a 512-point FFT and a 10 ms hop. The ASR branch stacks eight successive feature frames before projecting them to width 1,280. It runs 32 Transformer layers with ten attention heads, QK normalization, feed-forward expansion four and rotary position embeddings. Frame stacking changes the temporal rate from approximately 100 frames/s to 12.5 embeddings/s, or 80 ms per position. These are continuous vectors occupying LLM positions, not discrete audio-codec tokens, words or a promise of 80 ms recognition latency.³²

Component diagram, not a timing benchmark. Encoder windowing is explained separately below; all retained audio positions still reach one language-model sequence.

Sortformer estimates activity for up to eight speaker slots. Its weights remain frozen during speech-model training; where reference speaker segments exist, training can use those instead of predicted activity. The implementation pools/aligns speaker activity onto the ASR grid, thresholds it at 0.5 and applies a learned LayerNorm over the speaker dimension. A fixed sinusoidal speaker kernel then maps the eight slot indicators to acoustic width. For batch size $B$ and $T_a$ retained audio frames, ASR states $H$ and aligned activity $P$ are fused as follows; batch/time axes are implicit only in the kernel multiplication:

The saved speaker-kernel scale is 1. The kernel's sinusoidal rows encode speaker slots, not time or verified identities. The sum preserves $T_a$ and width 1,280; it does not append eight extra input tokens per frame. This gives the LLM a representation of acoustic content plus who was active, but it does not separate overlapping voices into clean waveforms or make speaker attribution infallible.⁴³

The identity connector performs no further temporal downsampling. One learned linear projection maps the fused representation to the Nemotron 3.5 Lightning language width:

NeMo replaces the audio locator with these embeddings inside the user turn. There is no separate cross-attention bank that leaves the text context untouched: audio and text become positions in the same decoder sequence. Conceptually the model generates

This explains both the flexibility and the risk. Instructions can request verbatim speech, clean written text, terminology or a summary, but a fluent continuation is not necessarily faithful audio evidence. A summary and a verbatim annotation are different targets even when both are useful.⁵⁶

The projection is not sufficient by itself to teach a text LLM to hear. The report trains the ASR encoder, projection and language model jointly from the first supervised fine-tuning step, without an initial frozen-LLM alignment stage, then adds a short reinforcement-learning stage for transcription accuracy. Sortformer stays frozen. The timestamp adapter and optional external drafter are trained on top of the frozen target afterward. Task-specific instruction/target pairs teach which output convention to produce from the same acoustic representation.²

What the 52 backbone blocks actually do

The released language backbone interleaves 23 Mamba-2, 23 MoE and six attention blocks. These are 52 separate residual blocks, not 52 conventional layers each containing attention and an FFN. The language backbone has roughly 31.6B total and 3.5B active parameters. Perception, projection and the CTC adapter add approximately 0.8B; the shared MTP module adds about 1.3B checkpoint parameters. The complete checkpoint is approximately 34B parameters, despite the 30B-A4B shorthand.²³

Mamba-2 carries recurrent state. In a schematic per-head recurrence, omitting the surrounding convolution, gates and normalization, an input updates a state matrix instead of saving every past key and value:

Here the configured state is $64\times128$ per head, with 64 heads per Mamba block. The recurrence compresses history into fixed-size state; it is not a lossless recording or an unlimited transcript memory. Attention provides explicit access to earlier positions. Its six blocks use grouped-query attention with 32 query heads, two KV heads and head width 128: query projection is $2688\rightarrow4096$, while each key/value projection is $2688\rightarrow256$. Sixteen query heads share each KV head. This reduces stored KV elements but does not eliminate the attention calculation over the growing sequence.³⁷

MoE controls per-position FFN computation, not context compression. Each MoE block has 128 routed experts, selects six for a position and also runs one shared expert. For block input $h\in\mathbb{R}^{2688}$, the router matrix has shape $R\in\mathbb{R}^{128\times2688}$. In the reference implementation, sigmoid router scores receive a correction bias for selection; mixture weights come from the original scores, are normalized over the selected six and scaled by 2.5:

Routed experts use $2688\rightarrow1856\rightarrow2688$ two-matrix FFNs, $f_e(h)=V_e[\operatorname{ReLU}(U_eh)]^2$, with elementwise squaring; the shared expert's intermediate width is 3,712. The saved moe_latent_size is null: do not transfer Bolt's LatentMoE geometry to this checkpoint. Six selected experts do not mean the other weights disappear from device memory. These equations describe the reference computation; production kernels may fuse and dispatch it differently.³⁷

The MTP module contains an attention block and an MoE block, shared across two training prediction depths. Reusing it can draft more than two tokens during inference; MTP8 is not eight independently trained extra layers. The target model verifies drafts. This can reduce decoding steps, but it neither compresses the audio prefix nor enlarges the context.²

How one hour fits into 65,536 positions

For duration $D$ in seconds, the frontend contributes approximately $D/0.08=12.5D$ audio positions. Ignoring padding/rounding at the ends, a 30-second chunk gives 375 vectors and one hour gives 45,000 vectors, not 3,600 tokens. Let $N_p$ include instructions, chat delimiters, vocabulary hints and prior conversation, and let $N_y$ include every generated token, including reasoning and special tags:

This is budget arithmetic, not a measured maximum transcript length. Silence still occupies duration and positions unless the application explicitly removes it. Eight simultaneous speaker slots do not multiply the audio positions by eight, but transcribing several overlapping speakers can increase output length substantially.²⁸

An illustrative, not measured, hour with 150 spoken words/minute contains 9,000 words. At an assumed 1.5 text tokens/word, the transcript needs 13,500 tokens; adding 512 prompt tokens gives 59,012 total positions. That fits with 6,524 positions to spare. Fast speech, different tokenization/scripts, speaker/event tags, a glossary, JSON or a second translated transcript can consume the margin. Use the released tokenizer and representative output to estimate $N_y$; words/minute alone is not a safety check.

The model card's direct-ASR example allows 16,384 completion tokens, so that cap can truncate a transcript even when the total context still has room. Its reasoning preset requests 20,480 completion tokens: with 45,000 audio and 512 prompt positions, the reservation would be 65,992, exceeding the context by 456. Reduce the completion/reasoning allowance, shorten the input or split tasks; a runtime can reject an oversized reservation before generation. Turning reasoning on is not free, and is not the recommended default for transcription.¹

The inner language configuration declares max_position_embeddings: 262144. That is not evidence that the released speech model reliably handles four hours. The speech training/release boundary remains one hour and 65,536 total positions; changing a server flag does not establish generalization beyond it. Several audio inputs or follow-up turns also share one request's context. Conversation history is supplied by the application, not a persistent memory inside the service.³¹

Local encoder windows, recording-wide language context

The technical report describes independent 30-second post-feature chunks during training, with no encoder attention across their boundaries. The outputs are concatenated in chronological order before the LLM. A name mentioned near the start can therefore influence text generation about a later passage through the LLM, even though the acoustic encoder does not attend across the full hour. In that reported setup, speaker slot indices are local to chunks; maintaining consistent transcript speaker labels across the recording is the LLM's job, not guaranteed biometric matching.²

Chunking also limits encoder attention work. For $T$ audio positions divided into chunks of $w$ positions, the attention-pair count scales approximately as $(T/w)w^2=Tw$, rather than $T^2$. At $T=45000$ and $w=375$, that is 120 times fewer pairs for this encoder-attention term only, not a 120× model speedup. Fixed-size chunk attention is linear in total duration. FlashAttention already avoids materializing the full quadratic score matrix; the motivation here is reduced attention computation and the diarizer's short-context training, not a claim that FlashAttention needs quadratic stored activations.²

The serving path is not identical to that training diagram. The pinned NeMo vLLM plugin bypasses generic waveform chunking when the Parallel Expert Encoder is mounted. Its own online path keeps context-extended ASR windows, discards overlap outputs, concatenates core outputs and carries Sortformer's speaker cache between windows. In this source snapshot, absent overrides, defaults are 500 encoded core frames and 50 context frames on each side: about 40 seconds plus four seconds of left/right context at 80 ms/frame. These are implementation defaults, not a different advertised recording limit. Do not assume the saved encoder_chunk_size_seconds: 30 alone describes the executed vLLM path.⁹⁴

None of these windows resets the LLM context, removes the 45,000 retained audio positions or makes the model causal microphone ASR. Nor is windowing free: boundary acoustics, speaker consistency and output stitching still need duration- and speaker-aware evaluation.

The long-audio memory budget is not just the KV cache

With $L_a=6$ attention blocks, $H_{kv}=2$ KV heads, $d_h=128$, sequence length $T$ and $b$ bytes per stored value, the target backbone's KV payload is:

At BF16 ($b=2$), 45,000 audio positions alone require about 264 MiB of KV payload per sequence; at the full 65,536 positions it is 384 MiB. The 23 Mamba blocks additionally need $23\times64\times64\times128$ recurrent state values, or 46 MiB at FP32 per sequence. This latter term does not grow with audio duration in the simple recurrent-state view.³⁷

These are component payloads, not peak VRAM. They exclude model weights, Mamba convolution/replay buffers, MTP or DFlash state, audio features, retained CTC states, activations, allocator padding and workspaces. ReplaySSM changes state storage/recomputation; cache dtype and runner change the actual allocation. Concurrency multiplies request-owned state, and the six attention blocks still have growing prefill/decoding costs. See the KV-cache guide for the distinction between weights, cache and runtime headroom.¹⁰

The report lists checkpoint sizes of approximately 67.5 GB BF16, 35 GB FP8 and 21 GB NVFP4, not those numbers as deployment VRAM promises. NVFP4 is mixed precision: it applies to LLM attention/expert projections, while Mamba projections, the MTP module and ASR projections use FP8; several other components remain higher precision. Quantization does not turn every operation or state into four bits, and its accuracy impact must be checked on the actual language, acoustics and speaker workload.²

What the hour-long context enables—and when to split anyway

Recording-wide language context can reduce the need to stitch independently generated transcripts, preserve terminology across an interview and support questions that connect distant passages. Direct audio questions can also use acoustic evidence absent from a text-only transcript. These are architectural opportunities, not a guarantee that every distant dependency, omission or speaker change will be handled correctly.

For a dataset, preserve the session and segment offsets even when auditing clips separately. A recording-wide pass may help flag missing turns or inconsistent speaker tags; a bounded clip-level pass localizes corrections and permits retries. Reconcile boundary words and speaker labels explicitly, and measure the trade-off on the corpus. For deployment validation, stratify by duration and score early, middle and late spans separately, including omissions, repetitions, speaker errors, truncation status, first-token/completion latency and peak memory. Being able to accept a one-hour file is a capacity claim, not evidence of uniform accuracy throughout it.

Applications beyond plain transcription

Prompting before fine-tuning is a useful baseline. NVIDIA's medical adaptation case study reports held-out-speaker WER of 22.44% with its best base-model prompt and 12.17% after adaptation: a 45.8% relative reduction. It used 855 tuning clips and 2,764 held-out clips, a fixed scoring pipeline and domain-specific training data. LibriSpeech test-clean WER increased from 1.54% to 1.69%. These are that recipe's reported results, not a promised German-domain improvement or this blog's measurements.¹¹

Auditing existing audio–transcript pairs

For each pair $(x,r)$, distinguish four independent dimensions: recording usability, lexical agreement with speech, annotation style, and speaker/timing correctness. Clean audio can have the wrong label; noisy audio can have an excellent label. A faithful dialect recording may be valuable precisely because a general model finds it difficult.

Proposed workflow, not an NVIDIA benchmark. The figure contains no measured quality scores.

1. Preserve and inspect the original

Keep the audio hash, original transcript, source/license, speaker/session IDs and segment offsets immutable. Check decoding, sample rate, channels, duration and sample clipping. Add VAD/speech fraction or a separately validated acoustic-quality estimator if useful. These checks assess the recording; they do not decide whether the transcript matches it. Keep hard, noisy and overlapping examples in a separate category rather than equating them with corrupt files.

2. Generate a blind hypothesis

Run Transcribe on the audio without showing it the candidate transcript. Specify the language when known and the labeling convention: verbatim versus cleaned, digits versus spoken number words, preservation of code switching and speaker tags. Keep prompt, checkpoint, quantization, decoder settings and output status in the audit record. Truncated output, decode errors and unsupported tasks are abstain, not poor-quality labels.

For a German verbatim-label corpus, an initial prompt to validate on a development set is: “Transcribe this German recording verbatim. Preserve fillers, repetitions and language switches. Do not summarize or add missing information. Write numbers as spoken words.” Use a different declared convention if the dataset expects written numbers or cleaned text; do not select a prompt separately for each candidate just to make it agree.

A second, architecturally different recognizer such as Parakeet can supply another hypothesis for selected cases. Different models can still share training data and systematic errors; two agreeing systems are not two independent human witnesses.

3. Compute disagreement under a declared convention

Let $h$ be the blind hypothesis and $\mathcal{N}$ a fixed normalization/tokenization policy. Compare the candidate and hypothesis using edit distance:

When $r$ is trusted human ground truth, this is conventional WER of $h$. When $r$ is itself being audited, call it reference-normalized disagreement, not an estimate of the label's true error rate. It can exceed 1 because of insertions; an empty reference needs a separate silence/missing-label decision. NeMo Curator exposes GetPairwiseWerStage, but its example thresholds are not calibrated label-quality thresholds for this workflow.¹²

Store raw and convention-normalized comparisons, with a versioned normalizer. Do not erase negation, units, decimal values or meaningful names. Use CER or language-appropriate segmentation when whitespace word tokenization is unsuitable. Compare critical entities separately: one wrong dosage may matter more than several filler-word differences.

The following examples are illustrative, not model outputs:

4. Use alignment and audio questions as supporting evidence

Timing is a separate computation, not thousands of timestamp tokens required inside the transcript. The approximately 50M-parameter CTC adapter takes the unfused ASR branch states at width 1,280. A stride-two transposed convolution upsamples their 80 ms grid to 40 ms, followed by two Transformer layers with 16 heads and a projection to a dedicated 23,862-token vocabulary plus the CTC blank. After decoding, forced alignment maps the supplied generated word sequence onto these frame probabilities; native Sortformer activity further refines boundaries. Upsampling learns a finer prediction grid, not a higher-rate input recording. Neither the 40 ms model grid nor the 10 ms refinement grid is a measured timing-error bound.²³

The CTC head aligns a provided/generated word sequence to acoustic states. NeMo reuses detached ASR states after generation, so it need not rerun the audio encoder for timestamps. Its implementation also distinguishes CTC word boundaries from native Sortformer speaker-activity segments on a 10 ms grid. The latter, not gaps between aligned words, is the appropriate timing product for diarization evaluation.⁶

Successful forced alignment does not certify a transcript: a dynamic program can force incorrect words onto plausible frames. Conversely, a model limitation can make a correct transcript hard to align. Treat alignment failure or suspicious coverage as review signals, not a proof of wrong labels. The ordinary API does not thereby expose a calibrated probability that an arbitrary pair is correct.

Audio Q&A can help a reviewer ask whether speech is missing or an entity needs attention. If a second pass is shown the candidate, explicitly ask it to treat that text as untrusted data, distinguish unsupported/missing spans, and abstain when uncertain. Keep its findings separate from the blind hypothesis. Its self-reported confidence is not calibrated, and English-trained audio QA should not be assumed to be a validated German corpus judge.

5. Calibrate routing, not a magic quality score

Start with retain_candidate, review and quarantine, preserving reason codes such as format_only, entity_conflict, missing_speech, speaker_mismatch, decode_failure and uncertain. Quarantine is recoverable, not deletion. Retained candidates also need random human spot checks.

An audit manifest should keep separate columns for original_text, blind_hypothesis, normalizer_revision, raw_disagreement, normalized_disagreement, audio_flags, pair_flags, alignment_status, decision, reviewer and the model/runtime revision. Leave uncomputed fields null. A structured model response is evidence to validate and store, not permission to overwrite the source label.

Build an independently human-reviewed calibration set stratified by language, accent, source, duration, noise, overlap and transcript style. Split by speaker/session to reduce leakage. Choose thresholds for the desired precision of automatic quarantine, and measure false quarantines among valid pairs and retained annotation errors. Recheck each slice after model or normalizer changes. A low-disagreement teacher can otherwise remove difficult but correct accents and make the next training set less representative.

Do not mix this audit into the final benchmark, overwrite references with model predictions, or evaluate a fine-tune on its teacher-rewritten labels as if they were independent truth. An accepted correction should be a versioned new label with evidence and review provenance.

Granary v2: reuse the contracts, not an imagined one-model pipeline

Granary v2 and NeMo Curator provide useful building blocks. The released tutorials separate metadata extraction, ASR and text enrichment. The Qwen3-Omni ASR tutorial explicitly keeps the original label out of its ASR prompt, checks suspicious repetitions/phrases and retains fallback provenance. The text tutorial produces tn_raw, itn_raw and itn_clean for spoken-form verbatim, written-form verbatim and cleaned written-form targets.¹³¹⁴¹⁵

Two limitations are easy to miss. The text pipeline reads no waveforms; its QA/captioning stages operate on transcripts and cannot certify acoustic fidelity. And these checked-in recipes use multiple models, not Transcribe as an already validated drop-in pair-quality scorer. The metadata tutorial also documents private reference checkpoints required for exact reproduction. Reuse its data contracts and available stages; do not promise one-command parity with NVIDIA's entire training pipeline.

Serving choices that change the audit

The launch recipe uses vLLM 0.32.0 plus the NeMo Speech plugin. Audio limits must be raised for hour-long files, and the total prompt/output budget must fit. Preserve special speaker tags when needed, check completion status and validate any requested JSON against your schema. Do not assume that an OpenAI-compatible endpoint means every speech endpoint, parameter or timestamp extension is interchangeable.¹⁰¹

The default decoder limits are 600 seconds and 25 MB. The launch example raises them with VLLM_MAX_AUDIO_DECODE_DURATION_S=3600 and VLLM_MAX_AUDIO_CLIP_FILESIZE_MB=180. An hour of 16 kHz mono PCM16 is about 115.2 MB before headers; its decoded float32 waveform alone is about 230.4 MB of host memory. A float32 WAV is larger than the example file-size allowance even though its duration fits. Check compressed-upload size, decoded audio, base64/request overhead and reverse-proxy limits separately.¹⁰¹

The pinned serving documentation notes that a long recording's encoder windows run in the engine step admitting it, while already-decoding requests wait. Chunked LLM prefill is not automatically preemptible audio encoding. Separate long offline jobs from latency-sensitive traffic, and validate admission limits and batching under load rather than interpreting max_num_seqs as a promise that many one-hour requests fit.¹⁰

MTP and the separate DFlash2 draft are alternative acceleration paths. The launch card recommends starting DFlash2 at seven speculative tokens and reports that fifteen were slower for its tested ASR workload. At that source snapshot, DFlash2 disables CTC word timestamps. NeMo's newer generic runtime documentation describes broader timestamp support, including runner-dependent behavior: verify the exact checkpoint/plugin/runner combination rather than mixing launch commands with later defaults.¹⁶

For an audit benchmark, record GPU, actual checkpoint precision, audio durations, batch/concurrency, prompt, output length and whether alignment is included. Aggregate real-time factor $RTFx=\text{audio seconds}/\text{wall seconds}$ is a throughput measure, not one-call latency. No claim that “100 hours take a few minutes” is portable without those conditions. Likewise, local execution gives control of data placement but is not by itself a privacy, redaction or consent guarantee.

In particular, the technical report's serving curves use 30-second and ten-minute recordings on an RTX PRO 6000 Blackwell Server Edition with 96 GB, an optimized NIM stack, warm-up, bounded concurrency and no requested timestamps. NVIDIA explicitly distinguishes those optimizations from the release-day open-source integration. They are not one-hour latency measurements, and neither the shorter-input speed nor the private optimized stack should be extrapolated to the pinned public recipe.²

The useful result is a traceable review queue: which pairs are suspect, why, what evidence was consulted and how many valid examples the policy accidentally removes. Nemotron 3.5 Transcribe can supply rich evidence for that system. It should not be its unchallenged ground truth.

Primary sources