LFM2.5-24B-A2B: Biased Routing, Short Convolutions and Ten KV Caches

Liquid AI released LFM2.5-24B-A2B on October 13, 2026, with downloadable base and post-trained weights. It retains the predecessor's hybrid convolution/attention and sparse feed-forward architecture; the release adds a larger vocabulary, longer context and revised training. This article explains the released text-generation checkpoint, rather than treating the inherited architecture as a newly invented mechanism.¹

Two separate savings matter: four of 64 experts run per token in each MoE layer, and only ten of 40 layers retain a growing attention cache. Neither saving makes the complete model a two-billion-parameter memory allocation. All memory values below are calculations, not measurements from a GPU run.

The exact checkpoint

The audit uses model revision 1769516ac588c4c558089c94f27d96b0dc67b183 and Transformers implementation revision 28034545c35acac05793695bd5c878f66e6b544e, checked October 14, 2026. The latter is a code reference, not a claim that Liquid's benchmarks used that commit. The checkpoint identifies Lfm2MoeForCausalLM; its configuration records Transformers 5.12.1.²³

These are configuration facts except head width, which follows from 2,048 / 32, and state width, which is confirmed by the cache implementation. The first two layers are dense and convolutional. Choosing a token mixer and choosing a feed-forward implementation are independent decisions.²⁴

For batch size $B$ and sequence length $T$, a block preserves $[B,T,2048]$. It applies RMS normalization before its mixer, adds a residual, then applies another RMS normalization before its dense or sparse feed-forward network and adds a second residual.³

The router has two different scores

Flatten the normalized hidden states to $X\in\mathbb{R}^{N\times2048}$, with $N=BT$. A router matrix $W_r\in\mathbb{R}^{64\times2048}$ produces 64 logits per token. The implementation applies sigmoid independently to them. With use_expert_bias=true, a stored 64-element bias affects the top-four selection, but is excluded from the mixture weights.³

The checkpoint enables this normalization and sets the routed scaling factor to 1. In exact arithmetic, the weights sum to slightly less than one because of the epsilon; finite precision can round away that difference. A selected expert applies a gated SiLU MLP; each has three matrices containing $3\times2048\times1536=9,437,184$ weights. The runtime representation can pack its two input matrices together; that packing does not change the function.²³

Here is an illustrative token with five candidate scores; the other 59 experts have lower selection scores. These are synthetic numbers, not checkpoint measurements.

E displaces D, but contributes using 0.40, not its biased 0.60. Using the selection scores as weights would implement a different model. So would replacing sigmoid with softmax: although both preserve logit order without bias, their normalized weight ratios differ, and adding bias after the activation can also change the selected set.

The inference code reads the stored bias; it does not adapt it to the current serving batch. Separating selection from contribution allows selection preferences without directly inflating output coefficients. It does not prove equal expert loads in production, nor reveal the training-time bias update rule. A batch may activate many or all experts across different tokens. Dispatch, token regrouping and uneven expert loads still matter; see expert parallelism.

A short convolution is a local mixer, not an embedding lookup

The convolution path projects a normalized hidden vector into three width-2,048 streams, called B, C and x in the code. To avoid confusing that stream name with batch size, call them $g,u,v$. Multiply $g\odot v$, run a depthwise causal convolution of width three, then multiply by $u$ and project back to width 2,048.³

The equation indexes kernel coefficients by lag; tensor-storage order may be reversed. The streams are direct linear projections, without a sigmoid on these multiplicative gates. “Double-gated” describes the multiplication before and after convolution. There is no cross-channel mixing inside the depthwise kernel; the surrounding dense projections provide it. This is neither Engram-style hashed retrieval nor an input-independent memory table.

The immediate receptive field is three positions. Stacked layers expand it, and the interleaved attention layers provide access to the full preceding sequence. A convolution layer alone cannot retrieve any arbitrary earlier token merely because the model accepts 131,072 positions. During ordinary cached decode, the implementation stores three recent $a$ vectors per convolution layer. The mathematical recurrence needs two preceding positions for its next output, but the runtime allocates a three-position state. Rollback recording can retain additional history.⁴

Full attention remains in ten layers

Each attention layer projects $Q:[B,32,T,64]$, $K:[B,8,T,64]$ and $V:[B,8,T,64]$. Four query heads share each KV head. Queries and keys receive per-head RMS normalization followed by RoPE; values do not. The configured RoPE is the default formula with theta $10^7$, not YaRN. The causal attention scale is $1/\sqrt{64}=1/8$.²³

$M$ is the causal mask. The expression is the logical operation; a fused kernel need not materialize its $T\times T$ score matrix. Full attention still has quadratic prefill arithmetic and a decode read that grows with cached length. Replacing 30 attention layers reduces those terms without eliminating them.

For BF16 keys and values, one sequence of $T$ cached tokens requires the following logical payload, excluding allocation padding, activations and workspaces:²

The hypothetical comparison keeps the same head geometry and dtype and gives every layer full attention; it is not a quality comparison with another released model. Ordinary convolution state adds $30\times2048\times3\times2=368,640$ bytes, or 360 KiB per sequence, independent of context length. Multiply payloads by independent concurrent sequences, accounting separately for shared prefixes and runtime layout. The KV-cache guide explains that distinction.

Active weights do not define resident memory

The weight index reports 23,971,585,280 parameters and 47,943,180,288 tensor bytes, approximately 44.65 GiB. That is serialized tensor payload, not peak device memory. Sparse execution skips unselected expert computations; it does not discard their weights from a fully resident deployment.⁵

Across 38 MoE layers, expert weights account for about 22.951 billion parameters. Replacing all 64 experts with four selected experts in this count gives approximately 2.455 billion active parameters, agreeing with the card's 2.45B figure. It counts all non-expert weights, including the tied vocabulary matrix, even though the input embedding is a row lookup. It is not an exact FLOP or memory-traffic estimate.⁵⁶

The tied vocabulary matrix alone has $128000\times2048=262,144,000$ weights: 500 MiB at BF16. Tying avoids a second matrix allocation, but the output projection still scores 128,000 vocabulary entries. Full BF16 tensor payload plus one maximum-length BF16 KV cache is already about 47.15 GiB, before all other allocations. Quantization or offload changes that budget and introduces its own accuracy, bandwidth and runtime constraints; see quantization.

Counting a multiply-add as two FLOPs, the selected expert matrix products cost approximately $2\times38\times4\times3\times2048\times1536\approx2.869$ GFLOPs per token. Computing all 64 experts would cost 16 times as much for that term. This explains the arithmetic saving while retaining a larger pool of trained expert functions; it does not establish a 16-fold end-to-end speedup or a quality advantage over a dense model. Attention, shared projections, the vocabulary head, dispatch and memory traffic remain outside that estimate.

What to verify in a deployment

The inspected Transformers implementation separates attention backend selection, prefill convolution and one-token convolution-state updates. It provides PyTorch convolution fallbacks and kernel substitution hooks; the default expert path groups tokens by expert and accumulates weighted outputs. A configured model name does not identify which optimized kernels actually executed.³

Liquid's published GPU test used one B300, SGLang 0.5.22, BF16, 1,024 input tokens and up to 256 output tokens, with continuously replenished requests at fixed concurrency. Its reported aggregate throughput is not single-request decode speed, a 131k-context test, or an isolated measurement of the router's benefit. No such benchmark was reproduced here.¹

Pin weights, tokenizer and runtime; verify the chat template and the actual convolution, expert and attention kernel paths. Measure prompt latency, per-request decode latency, aggregate throughput and task accuracy at your intended context and concurrency. The reasoning-only post-trained model can produce additional thinking tokens, so time to a usable answer also depends on output length. The card lists DSpark speculative decoding as coming soon, not a released speedup.⁶

The 131,072-position ceiling covers input and generated tokens together, including thinking tokens. A configured context limit alone does not establish reliable retrieval or task accuracy at that length.

The weights use LFM Open License v1.0, with a commercial revenue threshold; review the pinned license for deployment eligibility.⁷ This checkpoint generates text. Liquid d1-3B is a separate released decision model with shared question branches. A common family name does not make their serving paths interchangeable.

Primary sources