Ripple-65B: Embedding Value Residuals and Hybrid Attention

A released dense model, not a benchmark argument

Ripple-65B Preview is Ricursive Intelligence's released dense, text-only model. The instruction-tuned checkpoint has BF16 weights, an Apache-2.0 license and custom inference code; the publisher also provides a separate NVFP4 variant, base checkpoint and DFlash2 draft.¹ This article examines the BF16 instruction-tuned repository at d91aa65807bcfda98d964135230cd756be291774, checked on 6 November 2026.

The useful architectural detail is a value-residual path seeded by input embeddings, alongside PeriNorm, gated attention and a local/global attention mix. In the released HF code, this is not a direct reuse of the first layer's V tensors. The cache behavior also depends on the implementation, not just the configured window.² No model-quality, GPU throughput or peak-memory benchmark was run here.

Width, depth and the attention layout

The checkpoint has 80 layers, residual width 5,120, dense FFN intermediate width 40,960, 80 query heads, eight KV heads, and head dimension 128. The Q/output attention width is therefore 10,240, twice the residual width; the head dimension must not be inferred as 5,120/80. Ten query heads share each KV head. Vocabulary size is 100,352 and input/output embeddings are untied.³

Four local layers followed by one global layer repeat sixteen times: 64 sliding-attention layers and 16 full-attention layers. The local window is 1,024 positions including the current position. Local Q/K receive per-head RMSNorm and full-head RoPE with base 10,000. Global Q/K receive RMSNorm but no RoPE; their mask remains causal. NoPE is not bidirectional attention.³² Compare Beam's local/global positional split and the attention walkthrough.

Where the value residual actually enters

Let $e_t$ be the token embedding after the configured embedding scaling, and let $h_{\ell,t}$ be the current layer's pre-attention-normalized residual state. The pinned config sets the embedding multiplier to approximately 71.55417528, or $\sqrt{5120}$. The model takes its value-residual seed before entering the decoder loop and passes that same seed to every layer.³²

Each layer learns one scalar parameter $\lambda_\ell$. Q and K come from $h_{\ell,t}$, while the input to its own V projection is:

The orientation follows the code's lerp(seed, current, blend): a small $\alpha$ favors the embedding seed, a large one favors the current normalized state. Since $W^V_\ell$ is linear, this equals $(1-\alpha_\ell)W^V_\ell e_t+\alpha_\ell W^V_\ell h_{\ell,t}$. It does not equal mixing in $W^V_0 e_t$ from the first attention layer; every layer has its own projection.

This gives the value-content path direct access to a depth-stable lexical representation while queries and keys still depend on the contextual state. That is a dataflow explanation, not proof of improved recall or gradients. The scalar is layer-specific, not token- or head-specific. Its zero initialization implies 0.5 at initialization, not a measured interpolation coefficient in the released weights.²

For batch $B$ and current query length $T$, the reusable seed is [B,T,5120]. Projected Q is [B,T,80,128]; K and blended V are each [B,T,8,128] before transposition. The per-layer cache stores the resulting K/V for preceding positions. Sharing the seed across depth does not share those projected V caches across layers. During decode the seed covers the current input token, not another full copy of the past embedding sequence.²

Embedding scaling therefore affects both the main residual stream and the value seed. An engine port must preserve the convention: either apply the multiplier at runtime or fold it into embedding weights, not both. The reference explicitly handles both forms; this BF16 config carries the runtime multiplier.²

Two different gates, four block norms

Ripple's output gate is separate from $\alpha_\ell$. It projects the normalized attention input to 10,240 coordinates, evaluates a sigmoid (FP32 in this configuration), multiplies the concatenated attention output, then projects back to width 5,120. It is input-dependent and coordinate-wise; the value blend is one learned layer scalar.³²

The block uses PeriNorm, with separate pre/post RMSNorm around both attention and FFN:

These are four residual-width norms, in addition to the per-head Q/K norms. The FFN remains a dense SiLU-gated MLP. Before multiplication, the gate projection is clipped only above at 20, while the up projection is clipped to [-20,20]. This is not symmetric clipping of both projections or a SiTU-GLU definition.² All dense layers execute for each token; there is no MoE active-expert budget to substitute for the full weight budget.

A window mask is not cache eviction

In a local layer, the reference mask allows query position $p$ to read only keys with $p-1023\leq j\leq p$. But the K/V update occurs before that mask, and the reference creates DynamicCache() without a model configuration when it needs a new cache.²

Transformers v4.57.6, the version named in this checkpoint config, makes that unconfigured cache grow with full DynamicLayer tensors, appending K and V. It does not infer local-window storage from the mask. Config-aware sliding caches are a different path in that library.⁴ This is a static analysis of the reference's own cache-construction branch, not a claim that every engine or caller uses it.

Do not fix this by blindly swapping in a cropped cache: the reference mask constructs key positions from zero and compares them with absolute cache positions. A compact implementation must also preserve window offsets and causal semantics. No backend patch is proposed here.

For one request in BF16, define $S$ as retained context length. An ideal compact implementation with full global history and bounded local history has nominal K/V payload:

The leading factors are K and V, eight KV heads, 128 dimensions and two bytes/value. At 1,048,576 positions, that is 64 GiB global K/V + 256 MiB local capacity = 64.25 GiB. In contrast, retaining all positions in all 80 layers gives 320 GiB. Neither includes weights, allocator padding, temporary GQA expansion, masks, logits, draft state or workspace. These are derived payloads, not measured allocations; a windowed runtime may retain the current token separately or use different page granularity.

The global part still grows linearly with $S$. Four local layers per global layer do not make one-million-token inference constant-memory. See the general KV-cache budget.

Context and serving boundaries

The model card advertises up to 1Mi tokens; config metadata contains max_position_embeddings = 2,097,152. The supplied B200 serving recipe defaults to 1,048,576 total prompt-plus-output tokens. Keep these distinct rather than silently promoting a config value into a tested deployment limit.¹³⁵

That recipe targets SGLang 0.5.19 on two B200 GPUs, with complete BF16 target and matching DFlash2 checkpoints. Its target context and draft window differ: 16 draft proposals are configured, but the draft's attention window is 16,384 tokens. Neither is a 16× throughput guarantee. The recipe is not instructions for substituting the separate NVFP4 target unchanged.⁵

The HF reference uses PyTorch SDPA and explicitly repeats KV groups for attention. Its local path constructs a query-by-key boolean mask; the LM head also projects all returned hidden positions to vocabulary logits. These temporary allocations can matter during long prefill even when a different serving engine stores a compact hybrid cache. A framework label alone does not establish a particular fused kernel or peak memory.²

What to evaluate in practice

The dense model and embedding-seeded value path are useful candidates for code-oriented deployments, but no causal quality advantage follows from the architecture equations. Compare the BF16 target, any quantized target and speculative configurations separately on the same prompts, concurrency, output limits and correctness criteria. Record end-to-end latency, prefill peak memory, steady-state cache growth and accepted draft length; verify tool-call parsing and long-context retrieval.

The lesson is precise: one embedding seed can feed distinct per-layer V projections without sharing KV caches, and a bounded attention read window requires a matching allocator to bound stored local history. Neither a benchmark ranking nor a window-size field establishes those implementation properties.

Primary sources