NVIDIA Groq 3 LPX: The SRAM Decode Engine and the Arithmetic NVIDIA Won't Do

At Hot Chips 2026 NVIDIA announced that Groq 3 LPX, the SRAM-resident decode engine it inherited from the December 2025 Groq deal, is in full production.¹²

The pitch is one sentence long: decode is memory-bandwidth-bound, so put the weights in the fastest memory that exists — on-die SRAM — and stop streaming them from HBM once per token. That sentence is correct. The rest of this guide is the arithmetic the keynote did not include: the bandwidth-per-FLOP ratio, the capacity cliff, what the 3,431-tok/s Artificial Analysis result actually measures, and what the 35x marketing number is measured against.

One thing to state up front, because the coverage will not: this is resold Groq silicon. NVIDIA took the LP30 design in the roughly $20 billion, non-exclusive licensing deal of December 24, 2025, hired founder Jonathan Ross, president Sunny Madra, and most of the engineering team, and put its own brand on a Samsung-fabbed chip. The Hot Chips presenter, Igor Arsovski, is Groq's former chief architect.³⁴¹

The rack, in numbers

Each LP30 chip carries about 500 MB of on-die SRAM and no HBM. A full LPX rack holds 256 of them across 32 liquid-cooled 1U trays in NVIDIA's MGX architecture, Vera-Rubin-compatible, scaling past 1,000 LPUs on the fabric:¹⁵

The architecture is Groq's, not NVIDIA's: deterministic, compiler-scheduled execution, no caches, no branch prediction, no out-of-order machinery. The compiler schedules the pipeline at clock-cycle granularity over fixed 320-byte vectors — the Tensor Streaming Processor design from the 2020 ISCA paper "Think Fast," presented at Hot Chips under the same title by Arsovski and Santosh Raghavan.¹⁵

Determinism has a second-order payoff NVIDIA is proud of but that nobody prices in: because the compiler can predict power draw cycle by cycle, the rack pre-orders current from its regulators ahead of demand, cutting voltage droop by more than 60% and overshoot by more than 70% versus an uncompensated load. A workload with no jitter is a power delivery system with no jitter either.¹

Bandwidth per FLOP: the number a GPU vendor will not print

Decode at batch size 1 is the least favorable arithmetic operation in computing. To emit one token you must read essentially every weight of the model once, while performing only ~2 × parameters FLOPs of compute. A 31B model spends ~62 GFLOPs to buy one token; the question is how fast you can move 31 GB to do it.

The ratio that governs decode speed is memory bandwidth per unit of compute. Compute it for both machines:

That is roughly 97x more bandwidth per FP8 FLOP for the LPU rack — and if you count Rubin's 50 PFLOPS NVFP4 instead, ~270x. NVIDIA printed neither ratio on any slide.⁵⁶

Per chip, one LP30's 150 TB/s is about 6.8x a Rubin GPU's 22 TB/s of HBM4. To deliver that from HBM you would need HBM running at clock rates that do not exist on any 2026 roadmap; ~8 TB/s of HBM3E on the prior generation makes the per-chip gap about 19x. This is the entire technical case, and it is real. A GPU at batch 1 spends most of each token's time waiting for weights to cross the HBM bus; the LPU spends that time computing.⁵⁶

The corollary is that the LPX has far more bandwidth than its compute can consume: at 315 FP8 PFLOPS over 38.4 PB/s, compute saturation needs 8.2 FLOP per delivered byte — about 4× reuse of each weight byte when a MAC yields 2 FLOPs. Decode reuses nothing — each weight is touched once, so the stream sustains at most ~24% of peak FP8. That gap is not waste; it is the whole point. But it does mean the FP8 figure is nearly irrelevant to what the rack does, which is worth remembering whenever the 315-PFLOPS number is put next to a GPU's.

The capacity cliff

SRAM capacity is punishingly smaller per chip: one LPU carries 500 MB against a Rubin GPU's 288 GB of HBM4 — a 576:1 capacity ratio. The entire rack's 256 LPUs total 128 GB of SRAM — less than half of one Rubin GPU's memory, in a machine that costs like a full rack.¹ Now do the sizing honestly, because the launch coverage got the central number wrong.

Gemma 4 31B — the model NVIDIA benchmarked — has 30.7B parameters, so its FP8 weights are about 31 GB, not the "62 GB" making the rounds. The 62 in Tom's Hardware's coverage is LPUs, not gigabytes: 31 GB ÷ 500 MB = about 62 LPUs, a quarter of one rack, with three quarters of the SRAM left over — plus the entire DDR5 tier unused.¹⁷ The launch benchmark therefore does not touch the capacity cliff at all, which is precisely why it was chosen.

The KV cache does not sit in SRAM either. Under the attention-FFN disaggregation mode (below), attention and its 100K-token KV cache stay on Rubin HBM, where capacity-for-context belongs; only the FFN weights live on the LPU fabric.⁵ So the SRAM is spent on weights alone — the right call, but a call that concedes the capacity point before the benchmark even starts.

The cliff arrives where the money actually is: trillion-parameter MoE models. A 2-trillion-parameter model at FP8 has ~2 TB of weights:

• 2 TB ÷ 500 MB = 4,000 LPUs = 16 racks, past the 1,000-LPU scale point NVIDIA quotes, with the fabric stretched over multiple racks.
• Tom's Hardware says the same thing from the other side: large MoE models land "in four figures of chips across several racks."¹
• Every expert layer boundary crosses chip-to-chip links at 350 ns a hop, and attention-heavy work still needs Rubin HBM.

NVIDIA's own split-mode numbers (3–5x over Rubin alone) are measured at exactly that scale — 2T parameters, 400K-token cached context — which tells you the company knows the single-rack regime is a demo and the multi-rack regime is the product.¹⁵ Whether the deterministic fabric holds its 350-ns promise across 16 racks is the single most important unverified engineering claim in the whole story.

Decomposing the 3,431 tok/s

Artificial Analysis measured the LPX at 3,431 output tokens per second on Gemma 4 31B reasoning with a 100K-token context — against about 870 tok/s for the next-fastest public endpoint, roughly 4x, the fastest result ever recorded for that model.¹⁸

Two caveats before the praise. The comparison ran on a private, pre-release Gemini-4-class endpoint served through Google Cloud; the methodology was the median of 50 sequential client requests at a concurrency of one. That is a single-user latency test — the regime LPX is built for — and not a throughput test, so it cannot be compared to batched serving numbers.¹

Now decompose why the gap is ~4x. Not because SRAM bandwidth is being saturated — it is nowhere close:

• The bandwidth ceiling is astronomical. Re-reading 31 GB of weights per token at 38.4 PB/s aggregate would allow ~1.2 million tok/s. Even a single LP30 re-reading only its own 500 MB slice per token at 150 TB/s supports ~300,000 tok/s. The measured 3,431 tok/s is ~0.3% of the bandwidth roofline.
• So decode on LPX is latency-bound, not bandwidth-bound. At 3,431 tok/s a token ships every ~291 microseconds. That budget goes to the ~62-chip weight pipeline, 350 ns fabric hops, the attention step running on Rubin HBM against a 100K-token KV cache, and the per-token activation crossings back to the GPU domain.¹⁵
• The 870 tok/s baseline confirms the theory. A single Rubin-class GPU reading 31 GB of weights once per token at 22 TB/s tops out at ~709 tok/s. The fastest public GPU endpoint measured 870 — within ~20% of that ceiling. That number is the HBM bandwidth wall, visible in production data. The LPX wins by not standing in front of that wall at all: bandwidth per token is effectively free when the weights never leave the die.

That is the honest decomposition: the GPU number is a bandwidth measurement, the LPU number is a latency measurement, and the gap between them is exactly the gap between HBM and SRAM. NVIDIA's on-stage "11,000 tok/s" self-report for the same model should be read the same way — vendor-measured, single stream, no third-party methodology attached.⁹

Three split modes, and AFD as silicon partitioning

NVIDIA is not selling the LPU as a GPU replacement; it is a decode co-processor bolted onto Vera Rubin NVL72, orchestrated by the Dynamo runtime with an LPU extension to CUDA, an FPGA bridging the synchronous LPU domain to asynchronous host and GPU I/O.¹⁵ Three deployment modes:

1. PD disaggregation — Rubin GPUs run prefill and build the KV cache; LPUs generate output tokens. The classic prefill/decode split, now across two silicon families instead of two process pools.
2. Attention-FFN disaggregation (AFD) — within decode itself, GPUs keep attention and the KV cache on HBM while LPUs run the feed-forward and MoE-expert layers, exchanging intermediate activations once per token.⁵ This is disaggregation promoted from a software scheduling trick (which vLLM and Dynamo already do within homogeneous GPU clusters) to a physical partition of silicon by memory locality: the part of decode that needs capacity (KV) lives on HBM, the part that needs bandwidth (FFN weights) lives on SRAM. It is the most intellectually serious of the three modes — and the one whose per-token activation traffic across the GPU/LPU boundary had better stay small, because it runs once per token — about 3,430 crossings per second at the AA-measured 3,431 tok/s (one every 291 µs).
3. External-draft speculative decoding — a small draft model on the LPU proposes tokens rapidly out of SRAM; the GPU verifies them in parallel, and only draft tokens cross the link.¹⁵ Speculative decoding moves the bottleneck into the acceptance rate; the LPU is a genuinely good drafter, since a draft model's weights fit trivially in 128 GB of SRAM. Futurum flags external-drafter spec decode as the key mechanism NVIDIA highlighted at Hot Chips.²

The 35x number, measured against 2024's silicon

NVIDIA's headline economics — "up to 35x higher inference throughput per megawatt and up to 10x more revenue opportunity" — deserve their own audit.⁵

• The baseline is the GB200 NVL72, not Rubin. That is Blackwell: two GPU generations old at the time of the claim, and a rack you would not buy for premium interactive inference in 2026 in the first place.
• Rubin NVL72 on its own already claims up to 5x over GB200. The two multipliers live on different axes, and the deck conflates them. The 5x (10x with LPX) is a revenue-per-megawatt claim; the 35x is inference throughput per megawatt. On the revenue axis LPX adds the last ~2x (10÷5); on the throughput axis NVIDIA's published Rubin-vs-GB300 generational gain is ~3x, which would leave LPX the bulk of the 35x. NVIDIA does not decompose the 35x anywhere — treat any split as unverified modelling.⁵
• "Up to" is doing heavy lifting: the workload is a 2T-parameter MoE with a 400K-token cached context at 400 TPS per user — the exact shape SRAM residency flatters. All figures are NVIDIA-measured modeling; no third party has reproduced the 35x or the 10x.⁵²
• The efficiency gain is bought with capacity you pay for but cannot use. Tokens-per-watt excludes dollars-per-gigabyte — the LPX's worst ratio. Futurum reports the rack quoted as high as $1 million, which sets how large the latency premium on tokens must be for the ROI to close.²

The pattern is the same one AMD's "34x" rode: a big multiplier against a stale baseline, at a workload cherry-picked for the new machine's strength. The difference is that LPX's underlying physical argument, unlike a pure ratio, survives the audit — see the bandwidth-per-FLOP table above.

Verdict: when LPX wins, when it does not, and what NVIDIA is actually buying

LPX wins when latency per user is the product: batch-1 agentic loops, coding assistants, voice, anything where 3,000+ tok/s to one user beats 30 tok/s to 100 concurrent users. For that regime, the bandwidth-per-FLOP math is not marketing — it is the first production design that takes the decode roofline literally, and the AA 3,431 tok/s says the latency claim holds.¹⁸

LPX loses when capacity is the binding constraint: models past a few hundred gigabytes at FP8, high-concurrency throughput serving where batching amortizes HBM just fine, and long-context attention — which its own architecture delegates to Rubin HBM. MoE scale is the open question: 16 racks and four-figure chip counts are where SRAM economics stop being a lever and start being a tax, and where the 350-ns fabric promise must hold across racks it was never demonstrated on.¹⁵

What NVIDIA is actually buying is defensive. The Groq 3 LPX is a response to Groq and Cerebras in the one segment — single-stream decode speed — where a general-purpose GPU is architecturally wrong, delivered eight months after a $20 billion licensing deal structured to avoid merger review (two US senators have already asked the FTC about it; the deal that reportedly pushed Rubin CPX off the GTC roadmap in March).²³¹ It keeps specialized-decode challengers from owning the premium interactivity tier while Rubin ramps, and it locks one more rack type into the Vera Rubin platform — with LP35 (NVFP4) in 2027 and LP40 (NVLink, replacing the Groq interconnect) in 2028 already on the roadmap.

The Hot Chips slides are silicon math. The business is platform strategy, executed in Samsung-made ex-Groq silicon and priced at a premium tokens-per-watt will not explain.