KITE: Scale the Model, Freeze the KV Bill — the Prefill Invariant arXiv:2609.27294 Actually Proves (and the Three Things It Does Not)

The most expensive thing a language model does per fresh token — in FLOPs, in bandwidth, and in the HBM bill that follows both — is not thinking. It is writing attention keys and values. KV state is produced by the model's own layers, so the standard coupling is brutal: grow the model, and every fresh prompt token drags more and wider layers through the prefill pass, writing a fatter KV cache at the write price your memory tier dictates. arXiv:2609.27294, "KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scaling" (Hu, Wei, Zhou, Zhou, Li, Chen, Li, Wang, Zhu, Zhang, Jiang — StepFun, submitted September 23, 2026), is an attempt to break that coupling at the architecture level: scale the model, freeze the KV bill¹.

The idea, instantiated as the Step Scale Transformer (SST), is a two-tower decoder. A small tower — the Prefiller — alone produces every layer's keys and values. A second tower — the Decoder — never writes KV at all; it computes its own token-local queries and reads the Prefiller's cache, layer by layer. All new capacity lives in the tower that reads, not the tower that writes. Since prefill KV comes only from the Prefiller, the paper's claim is structural: prefill-bill invariance. Bulk prompt processing anchors to the small tower's geometry no matter how big the reader grows.

That claim is worth taking seriously, and it is worth decomposing past the abstract, because the abstract's own headline — a 67B MoE SST with 2.15B active per decode token beating 47B and 63B comparators on training loss at comparable cumulative compute — is a proxy story, not a serving story. This guide verifies every number at the primary, runs the KV-pin identity and a fleet-scale counterfactual in Python, and then gives equal first-class billing to the three things the invariant does not buy: reader-writer co-training risk, loss-vs-capability, and decode-side attention FLOPs. In the tiering thread we have been building, this is the architecture-side counterpart to the memory-side question — what if the model itself stopped growing the hot data?

1. The architecture, and what is actually invariant

SST is not exotic at the layer level. Each tower is a standard Transformer stack — in the experiments, 18 layers each, hidden width 2,304, MoE feed-forwards (512 routed experts, 8 selected), the same sliding-window-plus-full SSSF attention pattern the whole model family uses, one shared embedding and one shared output head, untied². The trick is the division of labor. The Prefiller runs first over the full sequence and produces layer-wise K and V. Each Decoder block then computes only a query from its own hidden state and attends over the Prefiller's KV at the corresponding layer, under the original causal mask. The Decoder's residual stream, FFN/MoE work, and a parameter-free RMSNorm "entry bridge" (embedding plus final Prefiller state) are all token-local: given the Prefiller's outputs, one Decoder position never depends on any other Decoder position².

That last sentence is the load-bearing restriction, and the paper states it as the boundary condition of the whole paradigm: future KV production must not depend on historical Decoder states². The invariant is structural, not frozen weights — the Prefiller keeps training:

• Prefill. Only the Prefiller processes the prompt and writes the KV cache. The Decoder runs only the final prompt position (to emit the first output token); everything else it does is token-local, so intermediate Decoder positions are simply omitted. Bulk prefill compute equals the small tower's compute.
• Decode. Every new token passes through both towers — the Prefiller extends the KV cache, the Decoder reads it and predicts.
• Training. Both towers train jointly after expansion; gradients reach the Prefiller through the reused KV, so nothing is frozen².

So the invariant is precise: the prefill KV bill is pinned to the Prefiller's layers x kv_heads x head_dim, independent of reader capacity. The paper's own parameter accounting makes the separation explicit — only the Prefiller contains K/V projections in the effective architecture².

2. The claim, verified at the primary

The scaling ladder keeps tokenizer, data recipe, and training protocol identical, and counts cumulative training compute — both training stages for SST, since the model is upcycled from a 33.8B source (about 168B source-stage tokens plus 222.55B continuation tokens, 390.54B total)². Against it stand two classic Transformers trained from scratch in the same family: 47B (1.48B active per decode token, 443.16B tokens) and 63B (2.02B active, 334.62B tokens). All three endpoints sit at approximately 100% of the 47B's theoretical training-FLOPs budget (B_ref, about 5.17e21 FLOPs)². Verified results at the final checkpoints:

The SST Prefiller carries exactly the source model's 1.120B active body per token — bulk prefill runs on a 33.8B-model's worth of parameters while the full model is 67B². On inference, the paper uses an analytical proxy (active-body counts with a 75% prefill / 25% decode cost mix, normalized to 47B classic): SST lands at 0.933 versus 1.365 for 63B classic and 1 for the reference — 6.7% below 47B and 31.6% below 63B². The proxy is honest about its own limits: with OpenRouter's observed prefill-heavy mixes (67.4–76.7% uncached-input charge share across six frontier models), SST stays 1.3–7.9% below 47B and 27.7–32.5% below 63B, but the break-even against 47B sits at a 65.5:34.5 prefill:decode mix — get decode-heavy and below cost parity and the 67B SST is more expensive than the smaller baseline². The paper's own conclusion stamps the scope: no scaling law, no causal attribution per ingredient, no measured serving speedup².

3. The KV-pin identity, run

The KV-cache formula from our glossary is bytes per token = 2 x layers x kv_heads x head_dim x weight-bytes³. Under KITE, every factor except layers of the Prefiller falls out of the scaling path — kv_heads and head_dim belong to the small tower, and the reader's width never enters. Run the identity against the paper's Table 2/5 geometry (8 KV heads, head_dim 128, BF16)²:

# KV-pin identity: keys+values are 2 tensors x layers x kv_heads x head_dim x 2 bytes (BF16)
def kv_kib(layers, kv_heads=8, head_dim=128, bytes_w=2):
    return 2 * layers * kv_heads * head_dim * bytes_w / 1024
 
print(kv_kib(18))   # 72.0 KiB/tok — SST: pinned to the 18-layer Prefiller, FOREVER
print(kv_kib(20))   # 80.0 KiB/tok — 47B classic
print(kv_kib(22))   # 88.0 KiB/tok — 63B classic: attention state muscles with the model
 
# the paper's inference proxy, reproduced from Table 2/6 active-body counts (robust to 0.1%)
prefill_sst = 1.120 / 1.477      # bulk prefill = source-sized Prefiller
decode_sst  = 2.155 / 1.477      # both towers, per decode token
decode_63   = 2.016 / 1.477      # 63B classic is prefill-and-decode symmetric
print(round(0.75 * prefill_sst + 0.25 * decode_sst, 3))   # 0.933
print(round(1 - (0.75 * prefill_sst + 0.25 * decode_sst), 4))          # 0.0665 -> 6.7%
print(round(1 - (0.75 * prefill_sst + 0.25 * decode_sst) /
              (0.75 * decode_63  + 0.25 * decode_63), 4))              # 0.3161 -> 31.6%

That reproduces the paper's 6.7% and 31.6% from the abstract to the digit². But notice what the identity actually says: SST's per-token KV is 72 KiB — 18% smaller than the 63B comparator's 88 KiB — a side effect of the two-tower design (the KV-producing stack is shallower than the comparator's 22 layers), not the invariant itself. The invariant is that a future, wider SST with a 4,608-wide reader or more experts writes the same 72 KiB. A hypothetical same-family model scaled the classic way to 40 layers at 16 KV heads would write 320 KiB per token — 4.4x SST's bill, on every fresh token, forever. That gap is what the paradigm sells.

And to be exact about the second row of the identity: prefill KVs are only half the fresh-token compute. The paper's forward-FLOPs accounting puts 63B classic at about 5.149B and the SST Prefiller at about 3.087B FLOPs per token at training sequence length — the 40% prefill saving is real but bounded by the non-attention fraction (MoE and readout), which the reader tower reaches only at decode².

4. The fleet counterfactual: the residency term, frozen vs growing

The number that made waves in September 2026 was Matt Barrie's anchor day: roughly 44 agents, about 4B tokens, with roughly 3.0B of them — a 75% looping fraction — cache reads of the fleet's own context, and about 1B fresh (we decomposed the bill in the fleet economics piece)⁴⁵. In the paper's own framing, agentic traffic is exactly the input-heavy regime KITE targets: uncached input exceeds output 10–16x in token volume across the six OpenRouter models it samples². So run one fleet day — roughly 0.9B fresh input + 3.0B re-read context — under a 63B classic versus the SST:

# Barrie-shape fleet day: ~0.9B fresh input, 3.0B re-reads (75% looping), ~0.1B output
fresh, reads, out = 0.9e9, 3.0e9, 0.1e9
 
# forward FLOPs/token from the paper's theoretical accounting (App. C): prefill vs decode
pre_63, pre_sst = 5.148513e9, 3.087252e9    # bulk prefill: full 63B vs 1.12B-active Prefiller
dec_63, dec_sst = 5.148513e9, 5.410678e9   # decode: both towers + readout -> SST pays MORE
 
pf_63, pf_sst = fresh * pre_63, fresh * pre_sst
print(f"{pf_63:.3e}", f"{pf_sst:.3e}", "prefill saving:", round(1 - pf_sst/pf_63, 3))  # 0.4
dc_63, dc_sst = out * dec_63, out * dec_sst
print("decode penalty:", round(dc_sst/dc_63 - 1, 3))                                   # 0.051
print("day total:", f"{pf_63 + dc_63:.3e}", "->", f"{pf_sst + dc_sst:.3e}",
      round(1 - (pf_sst + dc_sst)/(pf_63 + dc_63), 3))                                # 0.355
 
# the term the abstract does not headline: per-token KV, frozen vs growing
kv_63, kv_sst = 88 * 1024, 72 * 1024        # bytes/token: 22 vs 18 KV-writing layers, from cell 1
print(fresh * kv_63 / 1e12, "TB KV written (63B)", "|", fresh * kv_sst / 1e12, "TB (SST)",
      "|", round(1 - kv_sst / kv_63, 3))   # 81.1 vs 66.4 TB, -18.2%
print(reads * kv_63 / 1e12, "TB re-read traffic (63B) vs", reads * kv_sst / 1e12, "TB (SST)")
 
# residency: ~44 agents x ~50k tokens of live context parked in HBM
residency_tokens = 44 * 50_000
print(round(residency_tokens * kv_63 / 2**30, 1))    # 184.6 GiB live KV (63B)
print(round(residency_tokens * kv_sst / 2**30, 1))   # 151.1 GiB live KV (SST)

Four takeaways the arithmetic forces. (1) On this Barrie-shape day SST's estimated forward FLOPs are 35.5% lower in total — the prefill saving (40% off ~4.6e18 FLOPs) swamps the decode penalty. (2) Fresh KV written drops 18.2% (81.1 TB to 66.4 TB per fleet day), and — this is the invariant's real teeth — a future KITE expansion to 100B+ writes the same 66.4 TB, while a classically scaled model does not. (3) The 3.0B re-reads move 270 TB of KV for 63B classic versus 221 TB for SST across the day: every re-read touches the cache the writer tower sized, so the looping fraction inherits the discount. (4) Residency, the quantity our metering piece showed is the real commodity a shared server sells⁶, drops from about 185 GiB to about 151 GiB for the same 44 parked agents — and again, the frozen side of that term is the forward-looking claim: the residency cost of parking an agent stops growing with the model. These are identity-level estimates on the paper's own active counts, not measured serving numbers — the paper measured nothing at the serving level either².

5. Hype-breaker 1: the tower-separation bet

Here is what the abstract does not spell out, and what the paper's ablation cabinet conspicuously does not contain. The whole paradigm rests on an assumption with no direct test in the paper: that a fixed-ish size Prefiller can learn keys and values that remain good enough as reader capacity grows without bound. The reader is where all the intelligence growth happens — but every bit of it consumes the same writer's output. The paper never reports an ablation varying Prefiller size against a fixed Decoder, never runs a frozen-Prefiller arm, and never tests reversed or shared KV assignment beyond illustrating them as design options². Disclosure Figure 6 explicitly frames connectivity (KV assignment, hidden-state bridges, capacity split) as unexplored degrees of freedom, "not assumed to have equal cost or quality"².

The paper's own results, read closely, do document the co-adaptation cost, just without naming it as a risk. In §2.3/§3.2: gradients flow into the Prefiller through the reused KV throughout the joint stage — the Prefiller's activation ramps down and its representations keep adapting to serve the reader — with loss rising at conversion and then decreasing through continuation (Figure 4), the signature of a system re-fitting its writer after adding its reader. And the sanity case is in the comparators themselves: the 63B classic, which has 22 layers of KV-writers at width 2,816, hits 1.5921 with less total training (334.62B tokens) than the 47B baseline needs for 1.6006 at 443.16B — deeper stacks seem to write stronger KV per token; so maybe KV production isn't the layer-thrift afterthought the paradigm wants it to be. If, at frontier scales, the reader needs more from the writer — wider KV per layer, more heads, more layers — the Prefiller has to grow, the 72 KiB line breaks, and scaling recouples, resetting the KV-pin identity and — critically — its cache-economics payoff: the value of freezing 72 KiB/token depends entirely on 72 KiB being the endpoint. The paper neither confirms nor refutes this; it does show the current reader, a 2.155B-active Decoder, is satisfied with a 1.120B active writer at 390B tokens of training. Whether that holds at the next expansion step is the entire bet.

6. Hype-breaker 2: the loss ladder is not a capability ladder

"Lower training loss at comparable cumulative compute" is a clean scaling-ladder claim, and this paper deserves credit for counting both training stages — the upcycled route does not get to hide its source-stage tokens². But the reader should be precise about what sits on the ladder. Training loss is a consistency-of-fit metric, not a capability metric; the difference in question is 0.0106 versus the 47B and 0.0021 versus the 63B on EMA-200 at endpoint — a 0.13% relative edge on the latter². Holding out a candle: the paper does also report downstream tasks, and there SST outperforms both baselines across all seven (OpenBookQA 75.00 vs 71.00/68.50, MMLU 60.54, GSM8K 58.38, MATH 34.36, HumanEval 37.80, MBPP 51.40, BBH 52.74 — plus held-out arXiv NLL of 1.4421, lower than both)². These are standard open benchmarks at one checkpoint each; no error bars, no serving measurements, no long-context tests, no agent-loop evaluations — for a paper whose title contains "Agentic LLM Scaling," the evidence of agentic superiority is an OpenRouter charge-share table, not an eval². And the inference numbers are an analytical proxy on active-body parameter counts, which the paper itself twice stamps "not measured serving speedups"². Verdict as the scale of evidence goes: loss ladder solid, benchmark edge thin-but-consistent, agentic claim untested, cost claim analytical.

7. Hype-breaker 3: the reader still attends — decode FLOPs do not freeze

The invariant covers the prefill bill. It says nothing about decode, and the paper's accounting is refreshingly blunt about it: at a 75:25 mix SST is cheaper, but per decode token the SST costs more — every generated token passes through both towers (about 5.411B forward FLOPs/token versus the 63B's 5.149B in the paper's accounting, both towers' attention included, one shared readout)². The Decoder still runs full attention over the Prefiller's KV — reading the cache is not free. Break-even against 47B sits at the 65.5:34.5 prefill:decode mix; below that, in the paper's own Figure 5, the 67B SST is more expensive than the 47B baseline². For reasoning-heavy workloads — long thinking traces, low prefill share — the architecture's economics point the wrong way, and no KV-pin identity rescues that. The decode question is also where long-context reality bites: a reader attending over a 200k-token KV pays attention FLOPs that scale with context length, and nothing in the two-tower split reduces them — the HBF tiering arithmetic (bandwidth and FLOPs of re-reading large caches) applies unchanged. KITE freezes the write side; the read side is still on you.

8. What to actually conclude

KITE is a genuinely interesting architectural move, and one of the few 2026 scaling papers whose headline cost claim reproduces from its own tables by hand. The scaling ladder result is real: at matched cumulative training compute, a model whose prefill runs on a 33.8B-sized writer with a 2.155B-active reader reaches lower EMA-200 loss than both a 47B and a 63B trained from scratch, with modestly better benchmarks, and an analytical inference-cost proxy 6.7–31.6% lower in input-heavy mixes. The fleet arithmetic in section 4 is the steelman: in an agent-dominated world where three-quarters of tokens are self-rereads, an architecture whose KV residency stops growing with the model is worth real money.

Keep three labels on it. The tower-separation assumption — that a small writer keeps producing good enough keys for an ever-larger reader — is a bet the paper's ablations do not test; watch whether future KITE-family expansions report needing to grow the Prefiller. The capability evidence is a loss ladder plus seven small-benchmark deltas, with no agentic eval in a paper about agentic scaling. And all cost claims are active-parameter proxies, not serving measurements — decode costs more per token and break-even sits at 65.5:34.5. None of that makes the paper wrong. It makes it, like everything else in this genre, a claim about the training curve — with the serving story still to be run, and the reader's attention still to be paid.