LLM Inference Math: From Theory to Hardware

Large Language Models (LLMs) have become foundational in modern AI, but running them efficiently requires a deep understanding of the interplay between model architecture and hardware capabilities. Simply choosing the most powerful GPU isn't always the most cost-effective solution. The key is to know whether your workload is compute-bound or memory-bound.

This guide walks through the essential math to profile an LLM for inference, helping you select the right hardware and optimize its performance. We will apply these principles to a real-world example: running the Qwen/Qwen3-VL-32B-Instruct model on the powerful NVIDIA RTX PRO 6000 Blackwell Edition workstation GPU.

This article is inspired by the mathematical approach detailed in the Baseten blog post, "A guide to LLM inference and performance."

For a measured example, the MI350P vLLM optimization case study audits 118 observations and follows state-write reduction, K/V reuse and small-row routing. It distinguishes logical byte counts from hardware traffic and complete deployment results from isolated kernel timings.

Step 1: Understanding Your Hardware's Capabilities

The first step is to analyze the key specifications of our GPU. These numbers define the theoretical limits of our hardware. For the NVIDIA RTX PRO 6000 Blackwell Edition, the critical specs are:

• GPU Memory (VRAM): 96 GB GDDR7 ECC ¹
• GPU Memory Bandwidth: 1792 GB/s ¹
• FP16/BF16 Tensor Core Performance (dense): 503.8 TFLOPS ²

These three metrics—capacity, speed, and raw power—are the pillars of our analysis.

Step 2: Calculating the GPU's Operational Intensity (Ops:Byte Ratio)

A GPU's operational intensity, or ops:byte ratio, tells us how many computations it can perform for every byte of data it moves from VRAM. This is a crucial, hardware-specific ratio that reveals the balance between computation and memory access.

The formula is straightforward:

ops:byte Ratio = Compute Bandwidth (FLOPS) / Memory Bandwidth (Bytes/s)

Let's calculate it for our RTX PRO 6000:

• Compute: 503.8 TFLOPS = 503,800,000,000,000 FLOPS
• Memory: 1792 GB/s = 1,792,000,000,000 Bytes/s

ops_to_byte_ratio = 503,800,000,000,000 / 1,792,000,000,000
                  = 281.1 ops/byte

This means for our hardware to be fully utilized, our application must perform approximately 281.1 floating-point operations for every single byte it fetches from VRAM.

• If our model performs fewer operations per byte, we are memory-bound.
• If our model requires more operations per byte, we are compute-bound.

Step 3: Calculating the Model's Arithmetic Intensity

Next, calculate arithmetic intensity for the whole forward pass, separately for prefill and decode. Attention contributes work, but it is not the whole model: dense weight matrix multiplications dominate the simple linear term.

We'll use the parameters for the Qwen/Qwen3-VL-32B-Instruct model ³:

• Sequence Length (N): 4096
• Model Dimension (d_model): 5120
• Number of Attention Heads (n_heads): 64
• Number of KV Heads (n_kv_heads): 8 (GQA)
• Dimension per Head (d_head): 128

For a 4,096-token prompt, the 33.357B-parameter checkpoint has about 66.7 GB of fp16 weights. The standard linear-layer approximation gives:

Linear FLOPs ≈ 2 × 33.357B parameters × 4,096 tokens ≈ 273.3 TFLOPs
Weight-read bytes ≈ 33.357B parameters × 2 bytes ≈ 66.7 GB
Linear arithmetic intensity ≈ 273.3 TFLOPs / 66.7 GB = 4,096 FLOPs/byte

This 4,096 FLOPs/byte counts linear-layer FLOPs divided only by weight reads. Attention adds FLOPs, while reading and writing activations adds bytes, so it is an illustration rather than a measured whole-model intensity. At batch-1 decode the same linear-layer ratio falls to about 1 FLOP per fp16 weight byte. Context-length-dependent attention reads add further traffic.

Step 4: Identifying the Bottleneck

Now we compare the two ratios:

• GPU Ops:Byte Ratio: 281.1 ops/byte
• Linear prefill illustration: 4,096 FLOPs/byte of weight traffic
• Linear batch-1 decode illustration: 1 FLOP/byte of weight traffic

The simplified 4k-token prefill has more FLOPs per weight byte than the GPU's 281.1-FLOP/byte ridge, so its linear layers can be compute-bound. Batch-1 decode has much less arithmetic work per weight byte and is typically bandwidth-bound when the weights reside in VRAM. Actual bottlenecks also depend on attention, batch shape, kernels and cache behaviour; an attention-only intensity cannot classify the entire inference workload.

Step 5: VRAM and Performance Estimation

VRAM for Model Weights

A ~32.8-billion-parameter model (the Qwen3-32B text stack; the VL variant adds a vision tower for ~33.4B total) at half-precision (FP16) requires:

VRAM for Weights = 33.357 Billion Parameters * 2 Bytes/Parameter ≈ 66.7 GB

VRAM for the KV Cache

The KV cache size per token is:

KV Cache per Token = 2 * num_hidden_layers * n_kv_heads * d_head * 2 = 262,144 Bytes/token (Qwen3-VL-32B uses Grouped-Query Attention with 8 KV heads instead of 64, so the KV cache is 8× smaller than a full multi-head layout would need.)

With our 96 GB GPU, after loading the ~66.7 GB model, we have:

Spare VRAM = 96 GB - 66.7 GB ≈ 29.3 GB

This allows for a theoretical batch size of:

Batch Size = floor(29.285 GB / (262,144 B/token × 4,096 tokens)) = 27 sequences (before runtime, activation and allocator reservations)

Estimating Performance

Time Per Output Token (Decoding Latency):

Time/Token ≥ Model Size (Bytes) / Peak Memory Bandwidth (Bytes/s) ≈ 37.2 ms/token (weights only, batch 1)

This is a ~26.9 tok/s bandwidth ceiling before KV reads and other work, not a measured expectation.

Time to First Token (Prefill Latency): For a prompt of 512 tokens:

Linear compute time ≥ (512 tokens × 2 × 33.357e9 params) / 503.8 TFLOPS ≈ 67.8 ms

Conclusion

Our analysis shows:

1. Bottleneck depends on phase: The linear part of a 4k-token prefill can be compute-bound; small-batch decode is usually bandwidth-bound.
2. VRAM for batching: The 96 GB card has a raw capacity ceiling of 27 concurrent 4k-token KV sequences after the 66.7 GB checkpoint. Real capacity is lower once runtime and activation memory are reserved.
3. Performance bounds: Weight streaming alone limits batch-1 decode to ~26.9 tok/s; linear compute alone sets a ~67.8 ms prefill floor for a 512-token prompt. Neither is a deployment forecast or full TTFT.

These calculations provide a solid foundation for understanding LLM inference performance and making informed hardware decisions.

Related

• LLM VRAM Requirements: A Mathematical Deep Dive — the memory axis of the roofline.
• Tensor Parallelism: Slicing the Silicon — splitting one model across GPUs when a single card is not enough.

References