Muse Spark Open: Norm-Preserving Gates Are Not a Norm Guarantee

AI disclosure: This article was prepared with generative AI assistance. Editorial responsibility: Florian Zimmermeister (flozi00). Legal notice and contact.

Meta released Muse Spark Open on October 15, 2026. Its public model card describes a multimodal, text-generating MoE with 816B total and 42B active parameters. The interesting low-level detail here is how its implementation controls signal magnitude: attention gates, routing weights and residual gates are three different operations, with normalization at several distinct points.¹²

This is a public implementation audit, checked on October 18, 2026, using Transformers revision d4804f9d76b8c071b993d813e9e87067b780cbee. The Hub API identifies checkpoint revision 0fb02478b02565b2ce7fc57e6051c91172774ac9, but direct access to its configuration returned an authorization error. We did not access gated files, inspect weight tensors or run inference. Dimensions explicitly attributed to configuration defaults below are not a direct audit of the saved checkpoint configuration.³⁴⁵

Documented dimensions, implementation defaults and three gates

The public card documents 62 language layers, width 8,192, 128 query heads, 16 KV heads of width 64, and eight selected experts out of 256. The implementation's configuration defaults use a 4,096-wide expert space and 4,096-wide expert intermediates; NVIDIA's public integration documentation independently describes the reduced-width expert path.²⁵⁶

A residual gate is therefore not an attention mask, an expert router or a request-dependent early exit. Each decoder layer has separate post-attention and post-FFN residual gates. The implementation computes each branch before mixing it with the residual; a small gate does not by itself skip that branch's kernels.⁴

Two normalization orders implement different functions

Define parameter-free RMS normalization on a feature vector $x\in\mathbb{R}^D$ as $R_\epsilon$ below, and let $D_\gamma$ be the diagonal matrix with entries $1+\gamma_i$. Muse's implementation uses both output-scaled and input-scaled variants.⁴

MuseSparkTextCenteredRMSNorm implements $C_\gamma$. “Centered” refers to the zero-centered gain parameterization $1+\gamma$, not subtracting the mean of the input features. MuseSparkTextInputScaledRMSNorm implements the other order, $I_\gamma$. Scaling before normalization changes the denominator and therefore the output magnitude relative to scaling afterward. For the same diagonal gain, the two real-arithmetic outputs are collinear, but generally not equal.

For example, ignoring epsilon and rounding, take $x=(1,1)$ and $D_\gamma=\operatorname{diag}(2,1)$. Then $C_\gamma(x)=(2,1)$, while $I_\gamma(x)=(2,1)/\sqrt{2.5}$. Moving a channel-wise scale across RMSNorm is not a harmless algebraic cleanup. The input-scaled implementation also explicitly rounds the scaled vector to its input dtype before the FP32 normalization reduction, so real-number equivalence alone would not establish bitwise equivalence.⁴

Attention: normalize each head, gate each coordinate

For hidden states $[B,T,8192]$, the documented geometry produces Q with shape $[B,128,T,64]$ and K/V with shape $[B,16,T,64]$. Eight query heads share one KV head. The inspected implementation applies weightless RMSNorm to Q and K, multiplies Q by qk_scale_factor, and uses attention scale $64^{-1/2}$. RoPE is conditional: the model supplies no rotary embeddings for layers configured with a zero RoPE theta.²⁴

After attention, o_norm normalizes each 64-wide head output. The result is flattened and multiplied by $\sigma(W_Gu)$, where $u$ is the normalized branch input. The gate projection produces 8,192 values per token, not merely 128 scalar head gates. An output projection follows; the decoder then applies another, 8,192-wide parameter-free RMSNorm before residual mixing.⁴

The head norm, dynamic coordinate gate, output projection and branch norm are not interchangeable. In particular, applying one norm to the flattened attention output instead of one norm per head changes which coordinates share a denominator. Applying the gate after $W_O$ would mix a different set of channels.

Latent experts: normalization happens before the weighted sum

The router reads the full-width branch input. Separately, the MoE path projects 8,192 → 4,096, applies output-scaled centered RMSNorm and dispatches tokens to selected experts. For the documented/default geometry, expert weight tensors have shapes $[256,8192,4096]$ for packed gate/up projections and $[256,4096,4096]$ for down projections. Each expert is a SwiGLU in the 4,096-wide expert space, not a separate full-width 8,192 → 4,096 → 8,192 FFN.⁴⁵⁶

Let $z=C_{\rm pre}(P_{\downarrow}u)$. The selected expert computes $f_e(z)$ below. Crucially, the implementation applies input-scaled RMSNorm to each expert's output before multiplying by its routing weight and accumulating. The same normalizer module/gain vector is reused across experts in this implementation; applying it per expert does not imply 256 separate learned gain vectors.⁴

FP32 router logits are multiplied by output_multiplier, then passed through sigmoid. A correction bias affects top-k selection only. Mixture weights gather the original sigmoid scores, divide by their selected sum plus $10^{-15}$, and apply routed_scaling_factor. The same configuration field name used elsewhere is not permission to omit the scale here.⁴

The weighted mixture must not be replaced by normalization after aggregation:

As a simple counterexample with zero epsilon, take equal weights and expert outputs $(10,0)$ and $(0,1)$. Normalizing individually produces mixture $(1/\sqrt2,1/\sqrt2)$. Normalizing their raw mixture instead produces $(10,1)/\sqrt{50.5}$. The directions differ. This is an inference-kernel correctness condition, not just a stylistic choice about where to put RMSNorm.

Residual gates: derive the advertised intuition carefully

For a residual vector $r$ and a post-normalized branch vector $b$, MuseSparkResidualGate stores a learned vector $g$. The temperature $\tau$ comes from the configuration (default 0.3); its constructor's floor is $\delta=10^{-3}$. All operations below are coordinate-wise.⁴⁵

In real arithmetic, $\sigma(-w_i)=1-\beta_i$. Where the floor is inactive, $\alpha_i=\sqrt{1-\beta_i^2}$ and $\alpha_i^2+\beta_i^2=1$. This is not a convex interpolation: the coefficients generally do not sum to one. It is also not ordinary $r+b$ residual addition. The code's clamp keeps $\alpha_i\ge\sqrt{10^{-3}}$; once active, even the sum-of-squares identity no longer holds exactly.

The variance intuition requires assumptions. Suppose $r_i,b_i$ have zero mean, unit variance and correlation $\rho_i$, with the floor inactive. Then:

Only when the covariance term vanishes does this become unit variance. RMS normalization does not make a residual and its branch uncorrelated, does not subtract their means, and does not ensure identical per-coordinate variances. “Norm-preserving” is therefore a coefficient-design intuition under a statistical model, not a guarantee that every forward pass preserves a vector's norm.

For an illustrative scalar gate $\beta=1/2$, $\alpha=\sqrt3/2$. Equal-norm orthogonal branch/residual vectors preserve squared norm; perfectly aligned vectors instead multiply it by approximately 1.866, while opposite vectors multiply it by approximately 0.134. Those are derived geometric examples, not measurements of trained Muse gates or activations.

The complete layer has two separately normalized residual updates

First the layer normalizes the input, computes attention, normalizes its output and applies the attention residual gate. It then normalizes that new residual stream, runs the latent MoE, applies the FFN's input-scaled output norm and uses a different residual gate. The FFN output normalization is outside the expert mixture and serves a different role from each expert's normalization.⁴

Consequently, a port that retains the router and matrix sizes but replaces either update with hidden += branch does not implement the same model. Nor does a port that removes the per-expert norm because a later branch norm exists. Similar-looking graph fragments have different reduction axes, gain placement and rounding boundaries.

What this means for serving and kernel work

The inspected Transformers implementation protects the residual-gate modules, router weights and correction bias as FP32 modules. Its normalization classes accumulate in FP32 and return the input dtype. That is not a blanket statement that every runtime stores every activation or the entire residual stream in FP32: NVIDIA's integration describes an FP32 residual stream, and a port must preserve the precision contract of the particular implementation it targets.⁴⁶

A kernel may fuse compatible projections, dispatch and reductions, but it must preserve per-head versus full-vector axes, per-expert normalization before mixing, static versus token-dependent gates, clamp semantics and relevant dtype boundaries. Exact transformations that fold a fixed gain into a neighboring linear map require the actual surrounding graph; they do not justify moving an input-dependent RMS denominator through an expert sum. Even rewriting the gate's product as $1-\beta^2$ can change floating-point behavior near saturation.

These gates do not reduce KV-head count or eliminate resident expert weights. From the public attention geometry alone, separate BF16 K/V need $2\times16\times64\times2=4096$ bytes per retained token per attention layer per sequence, before paging, replication or other state. The retention policy still depends on local/global layer configuration and cache implementation; this article does not infer a whole-checkpoint cache budget from inaccessible configuration.

For a concrete contrasting expert layout, Nemotron 3.5 Bolt uses reduced-width routed experts but a full-width shared expert and different weight formats. Step-5-Preview illustrates the separate distinction between selective attention reads and retained KV storage.

The practical finding is a functional contract: Muse Spark Open's public implementation makes normalization placement and gate semantics part of the architecture. The equations explain their behavior and limitations, but establish neither better task accuracy nor faster inference. We performed mathematical checks only, not checkpoint parity tests, model evaluation or GPU benchmarks. Weight access, the Muse Model License and supported runtime revisions remain deployment prerequisites.²

Primary sources and audit boundary