NVIDIA Groq 3 LPX: The SRAM Decode Engine and the Arithmetic NVIDIA Won't Do

NVIDIA announced on August 24, 2026 that Groq 3 LPX is in full production.¹ The system combines SRAM-based LPUs with Rubin GPUs in several serving configurations.

The architecture aims to keep latency-sensitive working data in on-die SRAM. Low-batch dense decode can be constrained by repeated weight reads, while other workloads may be limited by attention, communication or compute. The rest of this guide is the arithmetic the keynote did not include: the bandwidth-per-FLOP ratio, the capacity cliff, what the 3,431-tok/s Artificial Analysis result actually measures, and what the 35x marketing number is measured against.

One thing to state up front, because the coverage will not: this is silicon based on the Groq design. NVIDIA took the LP30 design in the roughly $20 billion, non-exclusive licensing deal of December 24, 2025, hired founder Jonathan Ross, president Sunny Madra, and most of the engineering team, and put its own brand on a Samsung-fabbed chip. The Hot Chips presenter, Igor Arsovski, is Groq's former chief architect.²³⁴

The rack, in numbers

Each LP30 chip carries about 500 MB of on-die SRAM and no HBM. A full LPX rack holds 256 of them across 32 liquid-cooled 1U trays in NVIDIA's MGX architecture, Vera-Rubin-compatible, scaling past 1,000 LPUs on the fabric:⁴⁵

The architecture is Groq's, not NVIDIA's: deterministic, compiler-scheduled execution, no caches, no branch prediction, no out-of-order machinery. The compiler schedules the pipeline at clock-cycle granularity over fixed 320-byte vectors — the Tensor Streaming Processor design from the 2020 ISCA paper "Think Fast," presented at Hot Chips under the same title by Arsovski and Santosh Raghavan.⁴⁵

Determinism has a second-order payoff NVIDIA is proud of but that nobody prices in: because the compiler can predict power draw cycle by cycle, the rack pre-orders current from its regulators ahead of demand, cutting voltage droop by more than 60% and overshoot by more than 70% versus an uncompensated load. A workload with no jitter is a power delivery system with no jitter either.⁴

Bandwidth per FLOP: the number a GPU vendor will not print

Decode at batch size 1 is the least favorable arithmetic operation in computing. To emit one token you must read essentially every weight of the model once, while performing only ~2 × parameters FLOPs of compute. A 31B model spends ~62 GFLOPs to buy one token; the question is how fast you can move 31 GB to do it.

One way to compare the architectures is peak local-memory bandwidth divided by advertised compute. This is a specification ratio, not a measured speedup:

The specification ratio is roughly 97–111x higher for the LPU rack in this FP8 comparison. Comparing against Rubin's NVFP4 peak would mix precisions; neither ratio directly predicts token latency or throughput because the memory capacities, model partitions, interconnects, and cache behavior differ.⁵⁶

Per chip, one LP30’s 150 TB/s is about 6.8–7.8x the 19.2–22 TB/s range published on NVIDIA’s Rubin product pages. ~8 TB/s of HBM3E on the prior generation makes the per-chip gap about 19x. For a dense model at low batch size, weight traffic can limit a GPU. An LPU may avoid repeated HBM weight reads for weights resident in SRAM, but whole-system performance also depends on interchip communication and attention.⁵⁶

The simple peak-specification ratio also shows a low compute-to-bandwidth balance: at 315 FP8 PFLOPS over 38.4 PB/s, compute saturation needs 8.2 FLOP per delivered byte — about 4× reuse of each weight byte when a MAC yields 2 FLOPs. A single-stream dense decode has low weight reuse. The 24% figure is a simplified roofline calculation using peak bandwidth and peak FP8 compute; it is not a measured utilization ceiling for every LPX workload. This arithmetic does not determine measured FP8 utilization or whether a particular workload is compute-bound.

The capacity cliff

SRAM capacity is punishingly smaller per chip: one LPU carries 500 MB against a Rubin GPU's 288 GB of HBM4 — a 576:1 capacity ratio. The entire rack's 256 LPUs total 128 GB of SRAM — less than half of one Rubin GPU's memory, in a full rack system.⁴ That comparison describes memory capacity, not total system performance.

Gemma 4 31B — the model NVIDIA benchmarked — has 30.7B parameters, so its FP8 weights are about 31 GB, not the "62 GB" making the rounds. The 62 in Tom's Hardware's coverage is LPUs, not gigabytes: 31 GB ÷ 500 MB = about 62 LPUs, a quarter of one rack, with about three quarters of the nominal SRAM capacity left before runtime overhead and other state.⁴⁷ This rough weight-only calculation does not establish actual placement, DDR5 usage or the reason the model was chosen.

KV placement depends on the deployment mode. In NVIDIA’s attention–FFN split, Rubin GPUs handle decode attention and its KV cache while LPUs run FFN and experts.⁵ NVIDIA also describes SRAM as working storage for weights, activations and KV state in other LPU execution paths; one cannot infer the benchmark’s actual memory allocation from the public specifications.

The cliff arrives where the money actually is: trillion-parameter MoE models. A 2-trillion-parameter model at FP8 has ~2 TB of weights:

• If every FP8 weight had to reside in on-chip SRAM at once, 2 TB ÷ 500 MB would require about 4,000 LPUs, or 16 racks. This is a capacity thought experiment, not NVIDIA's disclosed placement plan; MoE activity, off-chip memory, quantization and model partitioning can change the requirement.
• Tom's Hardware says the same thing from the other side: large MoE models land "in four figures of chips across several racks."⁴
• Published chip-to-chip latency is a link specification; actual per-token interconnect and GPU–LPU overhead depend on placement and execution mode.

NVIDIA's 2T-parameter, long-context figures are vendor projections for a specified serving configuration, not a measured 16-rack deployment. The required memory placement and realized inter-rack latency remain to be established.⁴⁵ The cited 350 ns link latency must not be extrapolated to 16-rack end-to-end latency without measurements.

Decomposing the 3,431 tok/s

Artificial Analysis measured the LPX at 3,431 output tokens per second on Gemma 4 31B reasoning with a 100K-token context — against about 870 tok/s for the next-fastest public endpoint, roughly 4x in that reported comparison.⁴⁸

Two caveats before the praise. The 3,431 tok/s figure comes from Artificial Analysis testing an NVIDIA-hosted LPX configuration; the 870 tok/s comparator is the fastest public endpoint in that benchmark. NVIDIA's published graph does not disclose that comparator's exact hardware. The measurement concerns single-request output speed, not aggregate batched throughput. That is a single-user latency test — the regime LPX is built for — and not a throughput test, so it cannot be compared to batched serving numbers.⁴

A simple roofline calculation shows why the public benchmark cannot identify the bottleneck:

• The bandwidth ceiling is astronomical. Re-reading 31 GB of weights per token at 38.4 PB/s aggregate would allow ~1.2 million tok/s. Even a single LP30 re-reading only its own 500 MB slice per token at 150 TB/s supports ~300,000 tok/s. The reported 3,431 tok/s is ~0.3% of that idealized weight-only roofline, which ignores computation, data movement and attention.
• The measurement is far below a naive local-SRAM bandwidth roofline. At 3,431 tok/s, mean time per output token is about 291 microseconds. The published benchmark does not isolate how much of that time is spent in chip communication, compute, attention, or software.⁴⁵
• The 870 tok/s comparator does not identify an HBM bottleneck. A hypothetical single Rubin GPU streaming 31 GB of FP8 weights at 19.2–22 TB/s would have a roughly 619–710 tok/s memory-only ceiling, but the benchmark does not identify the public endpoint as a single Rubin GPU or even disclose its hardware and precision. The measured 870 tok/s therefore cannot validate that roofline model.

The benchmark establishes a roughly fourfold difference in observed per-user output speed for the compared endpoints; it does not by itself establish how much of that difference comes from SRAM bandwidth. NVIDIA's on-stage "11,000 tok/s" self-report for the same model should be read the same way — vendor-measured, single stream, no third-party methodology attached.⁹

Three split modes, and AFD as silicon partitioning

NVIDIA is not selling the LPU as a GPU replacement; it is a decode co-processor bolted onto Vera Rubin NVL72, orchestrated by the Dynamo runtime with an LPU extension to CUDA, an FPGA bridging the synchronous LPU domain to asynchronous host and GPU I/O.⁴⁵ Three deployment modes:

1. PD disaggregation — Rubin GPUs run prefill and build the KV cache; LPUs generate output tokens. The classic prefill/decode split, now across two silicon families instead of two process pools.
2. Attention-FFN disaggregation (AFD) — within decode itself, GPUs keep attention and the KV cache on HBM while LPUs run the feed-forward and MoE-expert layers, exchanging intermediate activations once per token.⁵ This is disaggregation promoted from a software scheduling trick (which vLLM and Dynamo already do within homogeneous GPU clusters) to a physical partition of silicon by memory locality: the part of decode that needs capacity (KV) lives on HBM, the part that needs bandwidth (FFN weights) lives on SRAM. Its viability depends on per-token activation traffic and synchronization across the GPU/LPU boundary — a per-token exchange whose volume and overhead depend on the partition. The 3,431 tok/s benchmark does not establish the crossing rate of this particular AFD configuration.
3. External-draft speculative decoding — a small draft model on the LPU proposes tokens rapidly out of SRAM; the GPU verifies them in parallel, and only draft tokens cross the link.⁴⁵ Speculative decoding moves the bottleneck into the acceptance rate; the LPU is a genuinely good drafter, since a draft model's weights fit trivially in 128 GB of SRAM. Futurum flags external-drafter spec decode as the key mechanism NVIDIA highlighted at Hot Chips.¹⁰

The 35x number, measured against 2024's silicon

NVIDIA's headline economics — "up to 35x higher inference throughput per megawatt and up to 10x more revenue opportunity" — deserve their own audit.⁵

• The baseline is the GB200 NVL72, not Rubin. That is a prior-generation Blackwell rack, so the quoted multiplier is not an LPX-versus-Rubin-only comparison.
• Rubin NVL72 on its own already claims up to 5x over GB200. The two multipliers live on different axes, and the deck conflates them. The 5x (10x with LPX) is a revenue-per-megawatt claim; the 35x is inference throughput per megawatt. On the revenue axis LPX adds the last ~2x (10÷5); on the throughput axis NVIDIA's published Rubin-vs-GB300 generational gain is ~3x, which would leave LPX the bulk of the 35x. NVIDIA does not decompose the 35x anywhere — treat any split as unverified modelling.⁵
• "Up to" is doing heavy lifting: the workload is a 2T-parameter MoE with a 400K-token cached context at 400 TPS per user — the exact shape SRAM residency flatters. All figures are NVIDIA-measured modeling; no third party has reproduced the 35x or the 10x.⁵¹⁰
• The efficiency gain is bought with capacity you pay for but cannot use. Tokens-per-watt excludes dollars-per-gigabyte — the LPX's worst ratio. Futurum reports the rack quoted as high as $1 million, which sets how large the latency premium on tokens must be for the ROI to close.¹⁰

The peak bandwidth-per-FLOP ratio is a hardware specification comparison. The 35x figure is a vendor scenario relative to GB200; neither establishes realized gain for another workload.

Verdict: where LPX may help and what remains unmeasured

LPX is aimed at low per-user latency in applications such as interactive agents. The reported 3,431 tok/s result supports high single-request output speed for the tested Gemma configuration; it does not measure cost, aggregate throughput or latency across other models.⁴⁸

Capacity and economics remain workload-specific. Models exceeding one rack’s SRAM require partitioning, other memory tiers or more racks; attention-heavy and high-concurrency workloads also depend on Rubin resources. The hypothetical 16-rack FP8 weight calculation is not a deployed benchmark. Published sources do not establish an end-to-end cost or latency advantage for those regimes.⁴⁵

NVIDIA positions LPX as a specialized decode complement to Rubin. The public material establishes that product positioning and the announced production status; it does not establish NVIDIA’s transaction motive or future competitive outcome.⁵⁸

A purchasing decision needs measurements for the chosen model, context length, concurrency, power and total system cost.