HBM Capacity and KV-Cache Economics

High-bandwidth memory (HBM) capacity affects how many long-context requests an inference server can keep resident. Supply constraints and allocation matter too. This guide separates a per-request KV-cache calculation, published GPU specifications, and forecasts about future memory configurations. Neither compute nor memory alone predicts realised serving throughput.

Calculate the KV budget first

For conventional multi-head, grouped-query or multi-query attention, an uncompressed KV cache has approximately

2 × layers × KV heads × head dimension × bytes per element × cached tokens

bytes per sequence. The factor two is for keys and values. Allocation blocks, sliding windows, prefix sharing, quantisation, other model state and serving overhead can change actual memory use. Architectures such as DeepSeek's multi-head latent attention require their own cache-layout calculation rather than blindly using this formula. See the KV-cache guide.

Here is a hypothetical 64-layer, 128-dimensional, two-byte-per-element comparison at 100,000 cached tokens:

These are decimal GB and arithmetic storage figures, not measured GPU occupancy or concurrent throughput. An idle agent whose KV state has been offloaded does not consume the same HBM as one whose complete cache stays resident. Prefix reuse and paging also make “one agent equals one full cache” an unreliable general rule.

Supply side: what the sources actually establish

Micron described an approximately 3:1 wafer-supply trade ratio for HBM versus DDR5 to produce the same number of bits, with the ratio expected to rise in later HBM generations.¹ This is an industry estimate, not a universal conversion for every process and product. TrendForce projected that HBM would account for roughly 30% of DRAM wafer input but 13% of DRAM bit supply among the top three suppliers by the end of 2027.² Those figures describe a forecast and corroborate the direction of the wafer trade-off; they are not a measured law of physics or a guarantee of 2027 output.

NVIDIA publishes up to 288 GB HBM4 per Rubin GPU in preliminary specifications.³ NVIDIA's rated HGX B300 is 270 GB HBM3E per GPU; GB300 NVL72 lists 279 GB.⁴ These products should not be grouped under a single “288 GB B300” specification.

TrendForce reported in August 2026 that NVIDIA was evaluating several Rubin Ultra memory-stack options and explicitly said the final specification had not been determined.⁵ A possible lower-capacity variant is a scenario, not an announced 192 GB Rubin Ultra SKU or a confirmed reduction from the 288 GB Rubin GPU. The earlier version of this article incorrectly presented that comparison as a product step-down.

A labelled capacity scenario

Assume, only for illustration, a GPU with 288 GB usable rated memory and 40 GB reserved for weights, activations and runtime; the remaining 248 GB could hold at most 248 / 7.0272 ≈ 35.3 complete 100k-token sessions if a particular model used 70,272 KV bytes per token. Under an otherwise identical hypothetical 192 GB GPU, the remaining 152 GB could hold 152 / 7.0272 ≈ 21.6 such sessions. This is a 39% reduction in the scenario's KV budget (152 / 248), not evidence of Rubin Ultra's eventual specifications or a measured request limit. Different weights, fragmentation, quantisation and offload policies change both numbers.

For deployment planning, measure the actual model's KV bytes per token and the server's reserved memory, then benchmark with realistic context lengths, cache sharing and concurrency. Capacity controls how many full contexts can reside; bandwidth, compute, scheduler and interconnect affect how fast they run. FP8 KV storage can approximately halve two-byte element storage when supported, but accuracy and implementation need testing. Offloading cold KV to host memory trades HBM capacity for transfer traffic and latency.

Sources and product status checked on 29 September 2026. Forecasts and scenarios are labelled as such.