Clef Omni: Audio, Video, Sparse Experts and Schema Decisions

Cloudflare released Clef Omni on 9 October 2026, with Apache 2.0 weights and a hosted Workers AI endpoint. It consumes a state, optional media and questions with explicitly allowed answers. Its output is a probability distribution over each question's options, rather than generated text. This article checks checkpoint revision 0db1cd2607d76a7bdb2a382f659e7b313079f84b against its configuration and reference code, using Transformers 5.10.2 as the implementation reference.¹²³⁴

The existing Clef and Clef-flash guide explains the two-stage decision head. Omni changes the backbone: it uses the Qwen3-Omni-30B-A3B-Instruct Thinker, including audio and vision encoders, a sparse mixture of experts and full causal attention throughout the text stack. It does not use the dense variants' three-linear/one-full attention cadence.⁵⁴

What actually runs

The loader sets enable_audio_output=False and retains the Thinker. The downloadable shards also contain the base model's speech-output modules, talker and code2wav, but those are excluded from this inference path. Input audio encoding remains enabled. Disabling speech output therefore does not turn Omni into a text-only model.²³⁶

These are checkpoint values, not specifications inferred from “30B-A3B.” The text configuration allows 65,536 positions; both the released helper's default input limit and the hosted context are 64,000 tokens. Schema, media and text share that budget.⁵⁷³⁸

Media become embeddings, not transcripts

The reference encoder places media before the text state, followed by the complete schema and an assistant suffix. It records spans for question instructions and option descriptions. If media, schema and fixed framing already exceed the token budget, encoding fails; otherwise it truncates the end of the text state to fit. A long attachment can therefore remove text evidence without shortening the schema.³

The processor expands media placeholders into the appropriate number of token positions. The Thinker replaces the embeddings at those positions with encoder outputs of width 2,048. It does not generate an ASR transcript or image caption as an intermediate step. Both speech content and other acoustic features can enter this path, but that architectural ability does not establish accuracy for a particular sound or event.⁹⁴

Audio: 16 kHz to roughly 13 positions per second

Encoded clips are decoded and resampled to 16 kHz mono. Raw sample arrays are taken as supplied, so callers must already provide that sample rate and layout. The feature extractor uses 128 Mel bins and a 160-sample hop: nominally 100 feature frames per second. Three stride-2 convolutions compress each 100-frame block to 13 positions; the audio encoder then projects its 1,280-wide representations to width 2,048.³¹⁰⁹⁴

For an unpadded Mel length $M=100q+r$, with $0\le r<100$, the published output-length function simplifies to the equation below. A 10-second clip with exactly 1,000 valid Mel frames contributes 130 audio positions, before framing tokens. Padding and boundary handling affect exact counts. This is a derived length example, not a measured throughput result.

Video: sampling, spatial merging and a common clock

The helper samples encoded video at two frames per second, scales frames to an approximate 262,144-pixel ceiling, and repeats the last frame when an even frame count is needed. Predecoded frame arrays must already follow the expected sampling cadence. The vision encoder groups pairs of frames, forms 16×16 spatial patches and merges 2×2 neighboring patches. For the processor's pre-merge grid $(T_g,H_g,W_g)$, the number of visual positions is:³⁵⁹

A resized 384×384 frame gives $H_g=W_g=24$, or 144 merged positions per temporal grid. At two frames per second and a temporal patch of two frames, that is approximately one such grid per second. This is a shape calculation; actual resizing, clip boundaries and marker tokens determine the request's final count.

For video with sound, the processor interleaves video and audio placeholders using a common temporal coordinate. At the default cadence, temporal grids are one second apart and their temporal positions advance by 13, matching approximately 13 audio positions per second. The implementation merges the streams in temporal order; it does not first transcribe audio and then attach text to frames.⁹⁴

There is a concrete restriction: use_audio_in_video is enabled only when every video in the record has a decodable audio track. One silent video makes the helper omit all video soundtracks in that record. Separate entries in audio still work. The collator also rejects a batch mixing video records with and without this audio mode.³

Synchronization here concerns the sampled, encoded representations. Two-frame-per-second sampling can miss brief visual events, mono conversion removes separate audio channels, and the reference helper decodes the full clip before inference. This release path is not a streaming decision interface.

Position encoding and visual features

The Thinker uses interleaved multimodal RoPE. Its rotary position tensor has temporal, height and width coordinates. Of the 64 rotary frequencies for a 128-dimensional head, 24 use temporal coordinates, 20 height and 20 width; height and width frequencies are interleaved with temporal frequencies. The configured RoPE base is $10^6$. Text positions use the same index on all three axes, while visual positions retain their grid coordinates.⁵⁴

Vision fusion also happens beyond the initial embedding replacement. Features from zero-based vision blocks 8, 16 and 24 are separately merged to width 2,048 and added at visual-token positions after text decoder layers 0, 1 and 2, respectively. These DeepStack additions reuse the visual positions; they do not append another three copies of the visual token sequence.⁵⁴

Sparse experts reduce FFN work, not resident weights to 3B

For one token representation $x\in\mathbb{R}^{2048}$, the router produces 128 logits, softmaxes them in FP32, selects the top eight probabilities and renormalizes only that subset. Each selected expert applies a SiLU-gated FFN with intermediate width 768. The routed output is a weighted sum:⁵⁴

The fused gate/up weight tensor has shape $[128,1536,2048]$; the down weights have shape $[128,2048,768]$. With no shared expert and a sparse block in every one of the 48 layers, the routed FFNs contain a derived 28,991,029,248 parameters. A token selects expert matrices containing 1,811,939,328 of those parameters across the stack. Attention, embeddings, routers and modality encoders are additional components; this is not a recalculation of the model name's total “3B active” figure.

All experts remain available to the router. The standard loader places the retained model on one device, rather than loading only eight experts. Cloudflare reports about 64 GB of BF16 backbone memory and testing on an H200 with PyTorch 2.11 and Transformers 5.10.2. That is the publisher's stated environment, not a measured minimum or a local measurement from this article.³²

Relative to evaluating all 128 experts, selecting eight reduces expert arithmetic per token. It does not promise a 16× end-to-end speedup: full attention, encoders, dispatch, matrix utilization and memory traffic still cost time. Nor does the router assign fixed “audio” or “video” roles to named experts; it routes each contextual token through learned scores.

Full GQA still has a prefill cost

For sequence length $L$, each text layer projects $[L,2048]$ into $Q:[32,L,128]$ and $K,V:[4,L,128]$, omitting the batch axis. Q/K undergo per-head RMS normalization and rotary encoding. Eight query heads share each KV head. The concatenated attention output is 4,096-wide and is projected back to 2,048; head width need not equal hidden width divided by query-head count.⁵⁴

Every text layer uses full causal attention. Prefill still requires quadratically many token-pair comparisons, even though an IO-aware attention kernel can avoid materializing the complete score matrix. Transformers dispatches through its selected attention backend and permits alternative expert implementations; the Cloudflare loader does not pin one particular accelerated kernel. Kernel availability and an actual serving benchmark are separate from checkpoint architecture.⁴³

Where Omni actually saves output work and memory

ClefModel saves the Thinker's lm_head weights for lexical option lookups and replaces the forward module with Identity. Consequently, the Thinker returns $H:[L,2048]$ in its normally named logits field, rather than computing $[L,152064]$ vocabulary logits. It also runs with use_cache=False and does not call generate. The fixed <think>/</think> suffix is input framing, not a generated reasoning trace.³

The KV-cache formula gives a useful counterfactual: retaining BF16 K/V for all 48 layers would take 96 KiB per token, or 5.86 GiB at 64,000 positions. This cache is not retained by the release wrapper. The formula excludes allocator and runtime overhead.

For another derived comparison, a BF16 $[64000,152064]$ vocabulary tensor alone would occupy 18.13 GiB. The wrapper avoids that projection and tensor. Its final hidden tensor plus the head's $[64000,1024]$ projected memory instead total 375 MiB at BF16. This is not peak VRAM: normalized copies, Q/K/V temporaries, attention and expert workspaces, encoder features and resident weights remain.

Skipping generation removes decode iterations and persistent decode-cache storage. It does not remove the multimodal encoders or the long full-attention prefill. The inference math guide explains why shorter output and smaller active FFNs cannot alone determine request latency.

From hidden states to bounded decisions

For one record with $F$ fields and $O=\sum_f O_f$ options, the head layer-normalizes $H$ and projects it to memory $M:[1,L,1024]$. It mean-pools the normalized question and option spans into 2,048-wide vectors, and averages the saved output-embedding rows for the option tokens. Their learned projections form option queries $[1,O,1024]$.⁷³

Two 16-head cross-attention layers let each option gather evidence from the complete memory. Each head is 64-wide. The routed options are pooled within each field; their summaries join the projected question, final sequence state and a type embedding. Four decoder layers then apply unmasked field self-attention, memory cross-attention and an FFN to $[1,F,1024]$. The backbone is causal, but this decision head can revisit the entire finished input.

Let $e_o$ be the lexical option vector, $q_f$ the pooled question and $g$ the final normalized sequence state. Let $u_f$ and $v_o$ be the final normalized field and routed-option vectors. The head combines a 2,048-dimensional lexical prior with a 1,024-dimensional joint scorer:

The positive learned scales $s_p$ and $s_j$ are capped at 100, and the residual MLP receives 4,096 features. The lexical path uses ordinary vocabulary rows; it is not an Engram n-gram memory or external retrieval. Attention-pair counts in the head scale with $2OL+4(FL+F^2)$ per head, excluding projection and FFN work. Declaring more fields or options therefore still increases cost.³

The fields share representations, but the API returns separate distributions for each field, not a normalized distribution over all combinations of answers. Joint attention does not guarantee logically consistent field choices. A choice selects the largest probability; noul exposes $p(\text{true})$; score returns $\sum_i i\,p(i)$ over the zero-based ordered levels. That last output is an expected index, not a measured physical quantity.

Deployment boundaries and evidence

As checked on 10 October, Workers AI provides @cf/cloudflare/clef-omni with the following hosted limits. They are API restrictions, not limits enforced identically by the Python helper.⁸

Local code also accepts paths, URLs, bytes and predecoded arrays. It requires Pillow for images and PyAV for encoded audio/video. Pin the checkpoint, processor, custom code and runtime together: settings such as the audio sample rate or video cadence determine the data entering the network. The reference forward re-collates and runs each record sequentially without padding, even when passed a batch. It is not evidence of continuous batching or parallel batch execution.³²

The pinned model card labels SGLang support as “coming soon.” Cloudflare's accompanying announcement describes SGLang work for Clef; that does not demonstrate a released Omni serving integration. The hosted endpoint and the published local reference are the verified paths here.²¹

Cloudflare describes frozen-backbone post-training with LoRA and a combination of label-smoothed cross-entropy and Brier loss. Those objectives train option selection and probability estimates; they do not guarantee calibration on unseen inputs. The published quality and latency results are vendor measurements, and this article does not reproduce them. In particular, text decision benchmarks do not by themselves validate audio recognition or synchronized audiovisual decisions.¹²

Omni's concrete advantage over the dense Clef variants is a direct path for audio and audio-bearing video to the schema head. Its sparse FFNs reduce the amount of expert computation per token, while avoiding output generation removes a different cost. Whether those mechanisms improve a deployment depends on media quality, input length, schema size, calibration and the selected serving implementation. More modalities alone do not establish better decisions on text or images.

Primary sources