Granite Speech 5.0 TurboCTC: Pyramidal Conformer Serving

ibm-granite/granite-speech-5.0-470m-turboctc is not a speech LLM. It is an encoder-only ASR checkpoint in the 470M parameter class with a Conformer stack and a CTC output. The current multilingual model supports English, French, German, Spanish, Portuguese, Italian and Japanese, uses a 16,384-unit BPE vocabulary, and is published under Apache-2.0.¹

This distinction matters in production: there is no autoregressive text decoder in the inference graph. All acoustic output positions can be evaluated in parallel, then greedy CTC decoding collapses repeated tokens and blanks. The price is a narrower product: transcription rather than a general speech-to-text assistant.

AI disclosure: This article was prepared with generative AI assistance. Editorial responsibility: Florian Zimmermeister (flozi00). Legal notice and contact.

1. The exact tensor path

The architecture paper and the released configuration agree on 16 Conformer blocks, hidden width 1,024, eight attention heads of width 128, a convolution kernel of seven, and attention chunks of 128 frames.²³ The implementation checked here is Transformers 5.16.1 at commit 93c8b7b485963a10800c91f55304db6be211c2bd; the checkpoint is fixed at 1dd23f46f6d4e6eae2531982ba2e33220086dd97.⁴ For batch size $B$, $T$ base 10-ms feature frames and stacked length $T_0=\lceil T/2\rceil$, the path is:

The first reshape halves the 100-Hz feature rate. The first two Conformer blocks each halve time again. The output therefore runs at 12.5 Hz, an aggregate factor of eight relative to the original log-Mel sequence.² A 60-second input has about 6,000 original feature frames, 3,000 projected frames, then approximately 1,500, 750 and finally 750 output positions.

The downsampling occurs inside the Conformer blocks. Their depthwise convolution uses stride two. Because the attention residual still has the longer sequence, adjacent residual frames are average-pooled before addition. The actual code uses unfold(1, 2, 2).mean(-1): an odd last residual frame is dropped, and the convolution output is cropped to match. Thus each strided block uses floor division, even though its padded convolution alone can produce a ceiling-sized result. The input processor instead pads a partial frame-stacking pair; these are different rounding rules.⁴

2. Why block attention changes the length term

Full self-attention has a score-pair count proportional to $T^2$ per head; fused kernels need not materialize the complete matrix. TurboCTC partitions time into independent chunks of $C=128$ frames, so the score work is approximately

which is linear in utterance length for fixed $C$. This does not reduce the $1{,}024\times4{,}096$ feed-forward matrices, but it prevents attention itself from becoming quadratic on long audio.²

For a 60-second example, the three sequence rates illustrate the effect. These are shape-derived counts, not measured latency:

At 50, 25 and 12.5 Hz, one 128-frame chunk spans about 2.56, 5.12 and 10.24 seconds. The progressively wider acoustic horizon partly compensates for the shrinking sequence. But block-diagonal attention also creates a boundary: frames in different chunks do not exchange attention directly in a given layer. Convolution and later layers can propagate information, but this is not global attention.

Also, chunk attention is not automatically a streaming API. The released Transformers path accepts complete tensors and masks; a deployment claiming stateful streaming still needs a tested chunk scheduler, overlap/context policy and latency measurement.

3. Intermediate CTC conditioning

Ordinary CTC treats token emissions as conditionally independent once the acoustic representation is fixed. TurboCTC feeds its own middle-layer belief back into the encoder:

Here $p^{(8)}_t\in\mathbb{R}^{16384}$, $A\in\mathbb{R}^{1024\times16384}$, and $c\in\mathbb{R}^{1024}$ is the injection bias. The result is the input to block 9, not an addition after block 9. The intermediate and final CTC heads share weights and bias. The paper's training objective is $0.2L_{8}+0.8L_{16}$; the released wrapper's ordinary forward(labels=...) computes final-head CTC loss only, so it must not be mistaken for a reproduction of that full training recipe.²⁴

This provides a token-posterior summary to the second half of the encoder without adding an autoregressive decoder. It does not remove CTC's alignment sum or turn the model into an LLM. In the authors' controlled English ablation, removing self-conditioning increased average WER from 5.83 to 6.00. That is a vendor experiment on the paper's checkpoint and test mix, not a guaranteed multilingual gain.²

4. The CTC math and parallel decoding

For target token sequence $y$, CTC sums probabilities over all frame-level paths that collapse to $y$:

The collapse operator $\mathcal B$ first merges adjacent repeated labels, then deletes blanks (ID 0 here). Order matters: a, blank, a becomes aa, not a. Greedy inference chooses $\hat\pi_t=\arg\max_k p_t(k\mid x)$ for every output frame in parallel, then applies $\mathcal B$. There is no token-by-token decoder loop and no decoder KV cache.

The wide head is still real work. A dense $1024\to16384$ projection has 16,777,216 weights plus 16,384 biases. If materialized in BF16, 750 logit rows for 60 seconds occupy 24.6 MB (23.44 MiB) per utterance. Self-conditioning also computes an intermediate vocabulary posterior; the final-logit payload alone is not a peak-memory estimate. Batch size scales that buffer linearly. Production runtimes should release or fuse logits promptly instead of retaining full posterior histories when only greedy text is required.

5. Kernel path: what was actually optimized

The authors replaced einsum-based content attention with PyTorch scaled_dot_product_attention. Reorganizing the operation allowed PyTorch's memory-efficient attention backend instead of the MATH backend. The paper explicitly reports that its Shaw relative-position-bias path could not use FlashAttention in that measured setup; this is a version-specific backend limitation, not a theorem ruling out every future fused implementation. They also replaced mathematically equivalent $1\times1$ Conv1d operations with Linear layers because the selected Linear kernels were faster.²

On one H100, CUDA 12.4, PyTorch 2.6.0 and batch size 128, the paper reports a combined 49% increase in inverse real-time factor (RTFx) for those two code changes—about 1.49× throughput, not 49% less elapsed time. All four compared variants used greedy decoding and the same pre-distillation checkpoint with 5.83% aggregate WER.² This isolates an implementation change; it is not a 49% promise for a different GPU, batch size, compiler or current multilingual checkpoint.

The released model is directly supported by AutoModelForCTC in Transformers 5.16 or newer and is also listed with MLX Audio support.¹ At review time, the pinned Hub revision was 1dd23f46f6d4e6eae2531982ba2e33220086dd97; the architecture fields checked in config.json include subsample_layers: [0, 1], context_size: 128, BF16 weights and the dimensions above.³

6. Training-only LLM distillation

The paper's English checkpoint used a temporary 75-million-parameter, four-layer decoder that cross-attended to the final encoder states and matched the top-ten-truncated token distribution of Granite 4.1 8B. That auxiliary decoder and the frozen 100K-token teacher head were discarded after training.²

Consequently, “distilled from an LLM” does not mean an 8B model is loaded in production. Serving remains the 470M-class Conformer plus its CTC head. It also does not transfer the teacher's open-ended generation abilities.

Be careful with dataset numbers: the architecture paper describes roughly 60,000 hours for the original English training. The October 21 multilingual model card reports roughly 153,000 hours for the current seven-language update.¹ Results from the English paper must not be silently relabeled as multilingual measurements.

7. Production decision boundary

TurboCTC is attractive when the workload is transcription, parallel batch processing matters, and a compact Apache-2.0 checkpoint is preferable to an autoregressive speech LLM. Its fixed-chunk attention and 8× time reduction directly bound the expensive sequence dimension; CTC removes the decoder loop.

Validate before deployment:

• input sample rate (16 kHz), channel mixing and very short/empty-clip handling;⁴
• WER by language, accent, noise, channel and domain vocabulary;
• end-to-end latency at the intended batch size, not the paper's batch 128 H100 setup;
• long-form chunk boundaries and any streaming wrapper;
• timestamp, diarization, contextual-biasing and beam-search requirements; and
• memory from activations and the 16K logit head, not parameter storage alone.

The model card positions the checkpoint for low-latency, high-throughput transcription. It does not make it a drop-in replacement for systems that need translation, dialog, speaker attribution or an integrated language model. Compare it with Parakeet FastConformer-TDT: both compress acoustic time, but TDT predicts token durations in a transducer loop whereas TurboCTC emits frame posteriors and collapses them with CTC.

Sources