Ling-3.1-flash: Grouped Routing and Two Different MLA Caches

AI disclosure: This article was prepared with generative AI assistance. Editorial responsibility: Florian Zimmermeister (flozi00). Legal notice and contact.

As of October 15, 2026, inclusionAI's Ling-3.1-flash has downloadable weights in 54 Safetensors shards, a custom implementation and serving instructions. The checkpoint combines seven Kimi Delta Attention (KDA) layers with one Gated Multi-head Latent Attention (MLA) layer per eight-layer group. Its useful engineering detail is that the presence of MLA does not by itself specify the cache layout: the released reference expands keys and values before caching, while an absorbed MLA serving path can retain their common latent representation.¹²³

This guide audits checkpoint revision a4ccd6b45c002ddb236f9cb77d52cc427482bf19 and SGLang revision f2b73d4f9b4e64415b8d775b0f811c7801f1728b. The calculations below are derived logical payloads, not GPU measurements. The existing Kimi K3 guide covers another released KDA/MLA design; Ling uses its own layer ratio, positional encoding and expert routing.

Configuration before headline numbers

The model card advertises a one-million-token context. The pinned config.json instead has max_position_embeddings: 262144, rope_scaling: null and RoPE theta 6,000,000. The pinned SGLang cookbook also identifies 262,144 as the context limit. A larger capacity claim is not an executable extension recipe or evidence of reliable long-context quality; this article uses the released 262,144-position configuration.¹⁴⁵

These values come from the configuration and layer-selection code. The first four layers retain KDA token mixing but use dense feed-forward networks. Later layers use MoE independently of whether their token mixer is KDA or MLA. The additional MTP layer is outside the 72-layer backbone and uses MLA in the reference implementation.⁴²

KDA replaces a token list with a recurrent matrix

Each KDA layer projects a $[B,T,4096]$ hidden stream into query, key and value streams of width $64\times128=8192$. Each projection passes through its own causal short convolution of width four with SiLU activation. After reshaping, each stream has shape $[B,T,64,128]$. The KDA kernel normalizes queries and keys, applies channel-dependent retention and a delta-rule write, then reads from a per-head recurrent matrix. An output normalization and sigmoid gate precede the projection back to width 4,096.²⁶

The delta write corrects the state's prediction for the current key toward the current value, rather than simply adding an uncorrected key/value outer product. The configuration enables the safe forget gate with log-retention lower bound $-5$. A bounded one-step decay does not establish lossless storage or arbitrary retrieval of the complete history. KDA's state has fixed shape with respect to context length; the interleaved MLA layers retain a sequence-indexed cache.

With one FP32 matrix of shape $[64,128,128]$ per KDA layer, 63 layers require 252 MiB per sequence. SGLang defaults to FP32 temporal state; configuration or environment overrides can change that dtype. This counts one live matrix per layer across the complete head set; convolution buffers, prefix checkpoints, speculative rollback state, allocator padding and replication are additional costs.⁷

MLA's expanded reference path

For each MLA layer, the query path compresses width 4,096 to 1,536, applies RMSNorm, then projects to $64\times192$ query components. Each head splits into 128 non-rotary and 64 rotary components. A separate projection creates a 512-component KV latent and a shared 64-component positional key. The normalized latent expands into 64 sets of 128 non-rotary key and 128 value components. RoPE is applied to the rotary query/key components; the positional key is shared across heads.⁴²

The reference concatenates the non-rotary and rotary keys and calls past_key_values.update with expanded $K:[B,64,T,192]$ and $V:[B,64,T,128]$. This is directly visible in BailingMoeV3MultiLatentAttention.forward. Reading only kv_lora_rank: 512 and assuming a 512-wide request cache would miss both this expansion and the positional key.

The attention score scale is $1/\sqrt{192}$. After attention, a separate projection of the current hidden vector produces one sigmoid gate per head; it multiplies that head's 128-component output before the final dense projection. This is an output gate, not a gate selecting cached tokens or experts. The configuration sets use_mla_nope: false, so the positional key remains part of the logical cache.²⁴

Why weight absorption keeps the same attention function

Let $c_t\in\mathbb{R}^{512}$ be the normalized latent at cached position $t$. For head $i$, split the latent up-projection into $W_{K,i},W_{V,i}\in\mathbb{R}^{128\times512}$. Let $r_t\in\mathbb{R}^{64}$ be the rotated positional key. For the current query, the two components are $q_i^N\in\mathbb{R}^{128}$ and $q_i^R\in\mathbb{R}^{64}$. The causal score is:

Associativity moves the non-rotary key up-projection from every cached token to the current query. Define $\widetilde q_i=W_{K,i}^{\top}q_i^N\in\mathbb{R}^{512}$. The same score follows from the concatenated query $[\widetilde q_i;q_i^R]$ and shared cached key $[c_t;r_t]$, each of width 576. After the same masked softmax, linearity moves the value up-projection after the weighted latent sum:

Therefore the cache can keep 512 latent plus 64 positional components per token per MLA layer, using the latent itself as the attention value. The head output gate still acts after the corresponding value projection. This is an algebraic reorganization of MLA, not an approximate token-eviction scheme. Finite-precision accumulation and quantization can introduce numerical differences; they do not justify claiming bitwise equality.

SGLang's Ling MLA class inherits the DeepSeek MLA implementation, including absorbed execution. Its MLA attention object uses one KV head, key width kv_lora_rank + qk_rope_head_dim, and value width kv_lora_rank; MLATokenToKVPool allocates the common latent/position buffer. Runtime dispatch can use different prefill and decode implementations, so temporary expanded tensors and workspaces must be budgeted separately.⁶⁸³

Two cache budgets for one checkpoint

Assume BF16 stored KV values, one sequence, all nine backbone MLA layers and one logical copy across the complete model. Neither formula includes KDA state, padding or workspaces. The absorbed representation needs no second independent value buffer: its value is a slice of the cached latent.

The ratio is approximately 35.56 for this KV term, not an end-to-end memory or speedup ratio. Different cache dtypes, tensor-parallel replication/sharding and page layouts change actual device allocations. Concurrent sequences and saved KDA checkpoints add costs; prefix reuse must restore the recurrent and MLA states at the same token boundary. See the KV-cache guide.

The table excludes the additional MTP layer. The reference gives that layer another MLA module; enabling speculative decoding also requires draft-cache/verification bookkeeping. A 72-layer backbone count alone is not a complete serving-memory budget.²

MoE selection first restricts groups

For a flattened hidden stream $X:[BT,4096]$, the reference router computes FP32 logits for 512 experts and applies sigmoid. Add the stored correction bias for selection, then divide the experts into eight groups of 64. Score each group by the sum of its two largest biased expert scores, retain four groups, and select eight experts from their 256 candidates. This is not unrestricted global top-eight selection.²

The mixture uses the original sigmoid scores, excluding the selection bias, and applies routed scale 2.5. Thus the routed weights sum to approximately 2.5, not one; the shared expert is added separately. Logical expert groups do not establish a physical mapping to eight GPUs, nor guarantee balanced production traffic. The LFM2.5 guide explains bias-only selection without this group restriction.

Activation details also matter. The configuration specifies clipping limits for routed experts in layers 65–71 and shared experts in layers 64–71. SGLang implements those limits in its expert/shared-MLP paths. Its shared path caps the activated gate above at 10 and the other multiplicative branch to $[-10,10]$. These hard clamps are part of the runtime function; a generic unmodified SwiGLU implementation should not be assumed equivalent.⁴⁶

The released custom reference MLP does not consume those per-layer clipping lists, while SGLang does. The cache derivation compares attention representations; it does not establish identical whole-model outputs across these engines.²⁶

Production boundaries

The checkpoint specifies block FP8 E4M3 weights with $128\times128$ scaling blocks and dynamic activation quantization, with explicit module exclusions. The Safetensors index reports 573,753,281,152 tensor bytes, about 534.35 GiB, across the released files. This includes checkpoint tensors and scale data; it is not a measurement of peak VRAM. The card's 560B-total/25B-active labels cannot replace inspecting actual stored tensors, MTP weights and runtime allocations.⁴⁹¹

The SGLang cookbook identifies H200 and GB300 recipes as verified while describing several other hardware recipes and prefill/decode disaggregation as unverified. It also disables prefill context parallelism for this KDA path. These distinctions matter more than treating every documented command as a tested production configuration. Pin the image digest/runtime and model revision, review custom code and dependencies, and verify cache layout, prefix-state alignment, speculative acceptance, latency and accuracy on the intended workload.⁵

No vendor speed claim or benchmark score is used here as proof of these mechanisms. No model inference was run for this article. The algebra explains the storage opportunity; source inspection identifies which paths implement it. A deployment still needs measured validation of the selected kernels, precision and context length.

Primary sources