Step-5-Preview: Sparse Reads Are Not a Compressed KV Cache

AI disclosure: This article was prepared with generative AI assistance. Editorial responsibility: Florian Zimmermeister (flozi00). Legal notice and contact.

StepFun released Step-5-Preview's weights on October 15, 2026, following its September API release. The model card describes a text/vision MoE model with approximately 600B total and 27B active parameters. This article examines one specific mechanism: its global attention layers search compressed proxy keys, then attend to selected original keys and values. Compressing the search representation does not compress away the original KV history.¹²³⁴

The audit uses the FP8 checkpoint and its bundled inference code at revision 364b3b03fed2477aa0953fe855a2c4c8c48ee3cc, checked on October 16, 2026. Tensor and memory calculations below are derived from that configuration and code, not measured throughput or peak VRAM. The KV-cache guide provides the baseline storage formula; Ling-3.1-flash illustrates a different mechanism, MLA weight absorption.

Read the layer types before counting memory

The 92-layer text backbone repeats three sliding-window layers followed by one sparse global layer: 69 local layers and 23 global layers. In the configuration, the latter retain the name full_attention; sparse_config.enabled and apply_to_layer_types make them sparse in execution. Disabling the sparse block would restore global dense attention, not a 512-token window.³⁵

The RoPE spans follow head_dim × partial_rotary_factors. The indexer has its own 32-dimensional rotary span. The misleadingly named yarn_only_types field does not establish YaRN here: the saved rope_scaling type is llama3, with factor 1.0. Read the actual scaling object and implementation, not the field name.³⁵

There are also three MTP layers outside the backbone, all sliding-attention layers. Consequently, the saved layer-type arrays contain 95 entries, not 92. The backbone has 88 MoE FFNs at indices 3–90, with eight of 352 routed experts per token and an additional shared expert. Attention selection and expert selection are separate operations; an expert budget does not describe KV storage.³⁵

Two paths leave the same hidden state

Let $N$ denote the packed tokens processed in this forward call, not the reserved context capacity. Ignoring tensor parallelism initially, hidden states have shape $X\in\mathbb{R}^{N\times4096}$. The main path projects $Q\in\mathbb{R}^{N\times64\times192}$ and $K,V\in\mathbb{R}^{N\times4\times192}$. It normalizes Q/K and applies the layer-specific partial RoPE.⁵

The indexer independently produces the tensors below. Its proxy keys are not the main attention's 192-wide keys. Four provider groups use the same compressed proxy-key store but compute different scores and selections; group $g$ controls the 16 main query heads that share KV group $g$.⁵⁴

Indexer queries use RMSNorm, keys use LayerNorm, and the 32-dimensional rotary transform applies to both. The compression logits remain a separate projection. No vocabulary lookup, Engram table or external retrieval index participates in this operation: these summaries are generated from this request's hidden states and belong to its per-layer cache.⁵⁴

Eight tokens become one search key, coordinate by coordinate

For completed block $r$ containing eight tokens, the kernel computes the following summary for each proxy coordinate $d$. A stable implementation subtracts a common maximum from the logits before exponentiation.⁴

This is not ordinary mean pooling, and not one attention weight per whole key vector. Each coordinate has its own weights. Although every coordinate stays within the corresponding keys' coordinate-wise range, the complete summary need not lie in the convex hull of the eight complete vectors.

The pinned kernel accumulates the reduction in FP32 and rounds the result through BF16 into E4M3 for storage. During decode it buffers the incomplete block's proxy keys and logits, producing its summary only when the eighth token arrives. Original main-path K/V are written independently and retained.⁴

Selection is a weighted ReLU score, not attention softmax

For query position $t$, provider group $g$ and historical block $r$, the implemented score is below. Here $\mathcal{Q}_8$ denotes the code's E4M3 rounding followed by lossless widening to BF16 for the score matmul. The four weights are signed linear-projection outputs with the prescale already included.⁴

The ReLU is inside the sum; signed weights mean the final score can still be negative. There is no softmax over indexer scores in this path. Despite sparse_indexer_softmax_variant: "ssmax" in the saved configuration, the reference registers ssmax_s only for checkpoint compatibility and does not use it in scoring. Replacing this code with a softmax-based indexer would implement a different function.⁵⁴

With zero-based token positions, only blocks $r<\lfloor t/8\rfloor$ compete for the top 512 historical slots. Equal scores are resolved by ascending logical block ID. The block containing $t$ is appended unconditionally, restricted to its causally visible positions. This is why “512 selected tokens” is incorrect: the historical budget is 512 blocks = 4,096 tokens, plus up to eight current-block tokens.⁴

The causal restriction is essential during prefill: a completed block summary may include positions later than an earlier query. Such a block cannot enter that query's historical candidates; its current block instead uses only original K/V up to the query. An unrestricted top-k over every precomputed summary would leak future information.

Main attention still reads original K/V

For a main query head $h$, let $g(h)=\lfloor h/16\rfloor$. Attention computes ordinary scaled dot products and a softmax over that group's selected original tokens, not over the summaries. The resulting head vector is sigmoid-gated before concatenation and output projection.⁵⁴

Two distinct approximations should not be conflated. Summary compression and indexer rounding can change which blocks are chosen. Restricting the main softmax changes its denominator and output relative to dense attention. Exact softmax on the chosen set is not proof of equality to full attention, and “retains the original K/V” is not proof of unchanged model quality.

What gets cheaper, and what stays length-dependent?

At $T=1,048,576$, the indexer searches about 131,072 block summaries, rather than a million individual proxy keys. The main path reads at most 4,104 original token positions per KV group for one query. The ratio $T/4104\approx255.5$ describes a token-read-count comparison, not an end-to-end speedup: search, top-k, projections, experts, communication, batching and kernel utilization remain.

For fixed dimensions, decode index scoring remains $O(T/8)$ and main attention uses $O(512\times8)$ selected positions. Scoring every prefill query still has an $O(T^2/8)$ term. This is not linear attention or constant-memory recurrence. The technical blog discusses grouping adjacent queries' overlapping selections for utilization; that optimization should not be assumed to exist identically in every reference or serving kernel.⁴⁶

Cache capacity: 4,104 reads do not mean 4,104 stored tokens

In the standalone implementation, every global layer reserves original K/V for the request capacity. At BF16, one token in one layer uses $2\times4\times192\times2=3072$ bytes for separate K and V. Local layers instead use 512-slot ring buffers. Let $T\ge512$ be the global capacity after rounding up to a 16-token page boundary. For one request, the unique, unreplicated backbone data payload is:⁵⁴

This is not a peak-VRAM estimate. It excludes weights, three MTP layers, incomplete-block buffers, page tables, lengths, RoPE tables, activations, score/selection workspaces, communication and allocator overhead. The reference also page-rounds global capacity to 16-token multiples. FP8 expert weights do not imply FP8 main K/V; the released SGLang recipes explicitly request BF16 KV.⁵⁴²

Nor is “unique payload” the sum across devices. In the pinned standalone TP=8 layout, each rank owns eight main query heads but caches one entire KV group. Each of the four global KV groups is therefore present on two ranks. All eight ranks also retain the shared summary store. The three components above consequently sum to $2\times(69\text{ GiB}+103.5\text{ MiB})+8\times736\text{ MiB}=143.952148$ GiB across ranks, before the excluded allocations. Do not apply this replication factor to a different engine or parallel layout without inspecting its cache implementation.⁵⁴

A configured million-token window is not a validated million-token launch

The standalone score_regions allocates a FP32 matrix shaped $[G_{\rm local}N,R]$, sized to live summaries rather than reserved capacity. At a one-million-token prefill, $N=T=1,048,576$ and $R=\lfloor(T-1)/8\rfloor=131,071$. At TP=8, $G_{\rm local}=1$, so this one tensor alone requires 511.996094 GiB per rank. This allocation is not a GPU measurement; it follows directly from the shape. It rules out naively running that standalone prefill on the documented H200 setup.⁴

The reference rejects a second prefill chunk on an existing cache and supports subsequent one-token decode steps. Its internal sliding-prefill chunking only bounds that path's temporary mask; it does not remove the sparse indexer's full score matrix. The limitation belongs to this pinned standalone implementation, not to the architecture in every possible engine.⁵

The release supplies separate SGLang/vLLM serving images. The SGLang recipes use --chunked-prefill-size 16384, BF16 KV and architecture-specific attention backends. The card distinguishes B300's FA4/MegaMoE recipe from H200's FA3/Triton recipe. A command requesting a maximum length or concurrency is not evidence that every request can fill that length simultaneously.²

The standalone README reports FP8 text/image generation validation on 8 H200 GPUs, TP=EP=8. Its BF16 full-model forward validation used 16 H200 GPUs, batch one and a 943-token image prompt, with reported peak PyTorch allocation of 76.37 GiB/GPU, excluding NCCL/CUDA-context allocations. Long-context workloads and throughput were not validated there; the complete BF16 file-shard conversion-to-generation workflow was also not run end to end. Those conditions do not establish a million-token production capacity.⁷

For deployment, pin both checkpoint and serving-image digest, inspect the supported sparse attention path, then test actual prompt lengths, output budgets, concurrency, prefix reuse and quality. The card's image tags are mutable names, not immutable runtime revisions. We did not run the model or reproduce the vendor's hardware tests. The custom StepFun Model License 1.0 also needs a use-specific review; downloadable weights do not imply an unrestricted open-source license.²⁸

The engineering takeaway is precise: compress the index, bound the reads, retain the selectable history. Step-5-Preview reduces the global attention working set without making its original global KV history constant-size. That distinction matters both when estimating memory and when choosing an implementation that can actually prefill the intended context.

Primary sources