Beam 1.0: Local RoPE, Global NoPE and Scalable Softmax

AI disclosure: This article was prepared with generative AI assistance. Editorial responsibility: Florian Zimmermeister (flozi00). Legal notice and contact.

Reflection made Beam 1.0's weights available on October 16, 2026, following its October 5 announcement. This is a released text-generation model, not a proposal: the instruction checkpoint has BF16, FP8 and NVFP4 variants under Apache 2.0. The model card reports 501.3B total and 23.4B active parameters.¹²

The interesting detail is its division of attention: 43 local layers use RoPE; nine global layers omit rotary position encoding and scale their queries with position instead. This article audits the BF16 checkpoint at 78f8f654d40a40919b8e9954b40dd879ecb9a692, Transformers at 54c0fe02b61062fe2666add21744c27035549644, and vLLM at 6af1e0ae273442d7cc19e60eed5b414ce299cb91, checked on October 19, 2026. Equations and byte counts below are derivations, not inference measurements.³⁴⁵

Two attention paths in one decoder

The checkpoint has 52 decoder layers, hidden width 5,120 and vocabulary size 151,936. Global attention appears at layers 6, 12, 18, 24, 30, 36, 42, 48 and 52, counting from one. All remaining layers have a configured window of 512. The saved maximum context is 1,048,576 tokens; the generic configuration class's default is not the checkpoint's context limit.³⁶

The 512-token window includes the current token: 511 earlier positions plus itself once full. Both the reference mask and vLLM's backend contract express that boundary. It is an attention visibility rule, not a promise that every cache implementation physically allocates exactly 512 slots.⁷⁵

Local layers build representations from nearby tokens; global layers can connect distant representations. That is a dataflow interpretation of the layer schedule, not evidence that a specific layer exclusively performs a linguistic task. Compare Mellum2.1's local/global layout, where global layers instead use YaRN.

Follow the tensors, not the parameter label

Let the normalized sublayer input be $X\in\mathbb{R}^{B\times T\times5120}$. Projections produce the following tensors before the head/sequence transpose. Matrix shapes use the PyTorch output-by-input convention.⁴

Attention width is 6,144, not 5,120. Six query heads share each KV head; GQA does not require storing six copies of K/V. Q and K receive RMSNorm along their 128-coordinate head axis, with epsilon $10^{-6}$, before the path-specific position operation. Values are not given that Q/K norm.⁴

For local attention, RoPE rotates normalized Q and K. For global attention, K stays unrotated and only Q receives the positive scale $a(p)$, where $p=t+1$ is the one-based absolute query position, not the number of newly supplied tokens in a cached decode call.⁴⁸

Bars denote the learned per-head-axis RMS normalization. The causal mask restricts the sum; it does not allow looking at future tokens. The attention result $O$ is then gated coordinate by coordinate, not by one scalar per head:

Only the additive bias is one value per head; the gate projection has 6,144 outputs per token. A sigmoid gate attenuates individual coordinates before heads are mixed by $W_o$. This is separate from MoE expert selection and does not remove heads from the KV cache.⁴⁵

NoPE does not mean position-independent. Global layers omit the explicit rotary transform, but their inputs already passed through local RoPE layers, the causal mask carries ordering, and $a(p)$ explicitly depends on absolute position. Replacing it with the sliding cache's current length would change the model.

What scalable softmax changes mathematically

At a fixed query and fixed visible keys, write $Z(a)=\sum_j e^{az_j}$ and $\pi_j(a)=e^{az_j}/Z(a)$. Softmax entropy in nats is $H(a)=\log Z(a)-a\mathbb{E}_{\pi(a)}[z]$. Differentiation gives:

Thus increasing $a$ sharpens a nonconstant fixed logit vector. It preserves logit order; it cannot turn an incorrectly ranked key into the best key. At positions 2,048, 65,536 and 1,048,576, Beam's scales are approximately 1.0718, 1.4186 and 1.8664. These are dimensionless scales, not speedups or quality scores.

The fixed-vector condition matters: extending a sequence adds keys and changes hidden states. The derivative does not prove that entropy falls as a real prompt grows, nor that the model retrieves equally well at every length. A simple counterweight is the following illustrative score model, not a measurement of Beam.

Suppose one relevant key beats each of $N-1$ equally scored distractors by a fixed margin $\Delta>0$. Its probability and the scale required to keep a chosen probability $r$ are:

More distractors increase the softmax denominator. Query scaling can counter that dilution, but Beam's prescribed power law is not fitted to this toy model's margin. At $\Delta=1$, $a=1.8664$ and $N=1,048,576$, the toy probability is only about $6.17\times10^{-6}$. This is not a Beam retrieval result: real margins, key correlations and representations are different. The useful conclusion is narrower: temperature control alone cannot guarantee long-context recall.

KV memory: nine growing caches, not 52

For batch $B$, sequence length $N$ and $b$ bytes per KV scalar, a useful window-sized logical payload budget is:

The factor two accounts for K and V. Every extra long-context token adds 18 KiB in FP8 or 36 KiB in BF16 across the nine global layers. The 43 window-sized local caches contribute approximately 43 MiB in FP8, independent of $N$ once full. For one sequence, before sharding or replication:

These are calculated tensor payloads with 512 local slots, not measured peak allocations. The pinned Transformers dynamic sliding cache retains 511 previous-token rows after an update; the returned attention tensors can include the whole current prefill chunk. Static caches, page rounding, backing-storage retention, temporary tensors and quantization metadata can differ. Model weights, activations, communication buffers and a speculative drafter are outside this budget.⁹

A hypothetical 52-layer all-global model with the same KV geometry would hold 104 GiB of FP8 KV at 1,048,576 tokens. Beam's architecture reduces that logical payload to about 18.04 GiB, but does not make attention linear-time: full-sequence attention work still scales as $O(9N^2+43NW)$, with $W=512$, omitting shared head-width factors. Flash-style kernels avoid materializing the full score matrix; they do not eliminate those global dot products. See the KV-cache guide and inference math.

The surrounding model and serving boundary

Layers 1–3 use dense SwiGLU with intermediate width 20,480; the other 49 use 256 routed experts of width 2,560, selecting eight per token with no shared expert. FP32 router logits become sigmoid scores. Group selection sums the two largest bias-corrected scores in each of eight groups, retains four groups, then selects eight experts. Mixture weights use the uncorrected selected scores, normalized and multiplied by 2.5. Each sublayer has pre/post RMSNorm and a depth-scaled residual addition.³⁴

Sparse activation is not sparse residency: 23.4B active parameters do not mean only 23.4B weights must fit. Expert placement, precision and offload determine the actual weight-memory layout. Nothing in the KV derivation measures the quality loss of FP8 cache or NVFP4 weights.

vLLM has a concrete NVIDIA serving path: it can merge Q/K/V/gate projections, fuse Q/K normalization with RoPE or SSMax and cache insertion, and apply a separate output-gate kernel. In this pinned implementation, fused cache insertion requires both Q and the cache to be BF16; the FP8-cache path instead passes K/V to the attention backend. Do not assume the same kernel sequence for both cache dtypes. The optimized operations explicitly preserve activation-dtype rounding boundaries. Floating-point fusion is therefore not justified merely by rearranging the real-number equations.⁵⁸

The release-day serving example targets the NVFP4 checkpoint on a GB300 node with TP4, FlashInfer attention and FP8 KV. It optionally adds the separately released DFlash2 drafter. That is a documented deployment configuration, not a guarantee for arbitrary GPUs or an assertion that this article ran it. Pin checkpoint, serving commit/container digest, tokenizer/template, cache dtype and parallel layout together; a moving main or Docker tag is not a reproducible deployment.¹⁰

Before production, validate long-context accuracy on your workload, prefill memory peaks, cache numerical effects and request concurrency. The saved million-token limit describes supported sequence capacity, not a measured universal recall guarantee. No GPU inference, quality benchmark or latency experiment was performed for this article. The result here is a source audit and a transparent memory/softmax derivation.

For related mechanisms, Step-5-Preview separates sparse global reads from retained original KV, while Ling-3.1-flash contrasts recurrent state with alternative MLA cache layouts. Neither is the same operation as Beam's temperature-scaled global GQA.

Primary sources