LLM VRAM Requirements: A Mathematical Deep Dive

Deploying Large Language Models (LLMs) requires careful consideration of GPU memory. This guide works through an illustrative inference estimate for Qwen3-VL-32B-Instruct and separately accounts for full-training model states.

How much VRAM does an LLM need?

For inference, budget model weights + KV cache + runtime allocations. The weight payload alone is approximately parameters × bits per weight ÷ 8 bytes. A dense 32B model therefore needs about 64 GB at BF16, 32 GB at 8 bits or 16 GB at 4 bits just for weights, before quantization scales, KV cache and runtime overhead. These are decimal GB estimates, not a guarantee that a model will fit on a card of that size.

Context length and concurrent sequences increase KV-cache memory. This guide calculates those components separately for Qwen3-VL-32B, then distinguishes inference from full training. To explore a model, GPU and context configuration directly, open the LLM VRAM calculator. For format overhead, see the quantization guide; for cache formulas, see KV cache explained.

Why VRAM Calculation Matters

Before deploying an LLM, you need to answer critical questions:

• How much GPU memory will my model consume?
• Can I run this model on my current hardware?
• Which quantization method provides the best memory-performance tradeoff?
• How many concurrent users can I support?

This guide provides the mathematical foundation to answer these questions accurately.

Example Model: Qwen3-VL-32B-Instruct

We'll use Qwen3-VL-32B-Instruct as our reference model throughout this guide. This multimodal model combines vision and language capabilities with the following architecture:

Configuration Source: The model configuration is extracted from the text_config section of the model's config.json file on Hugging Face Hub.

Core Memory Components

VRAM consumption for LLMs consists of four primary components:

1. Model Weights Memory

The base memory required to store the model's parameters.

Model Weights (bytes) = Number of Parameters × Bytes per Parameter

Bytes per Parameter depends on the data type (quantization level):

Example Calculation for Qwen3-VL-32B:

Number of Parameters: 33,357,390,064 (~33.4B)

float32:  33,357,390,064 × 4.0   = 133,429,560,256 bytes = 133.43 GB
float16:  33,357,390,064 × 2.0   = 66,714,780,128 bytes  = 66.71 GB
int8:     33,357,390,064 × 1.0   = 33,357,390,064 bytes  = 33.36 GB
int4:     33,357,390,064 × 0.5   = 16,678,695,032 bytes  = 16.68 GB

2. KV Cache Memory

The Key-Value cache stores intermediate attention states for efficient autoregressive generation. This is the most significant dynamic memory component during inference.

KV Cache (bytes) = 2 × Batch Size × Sequence Length × Num Layers × Num KV Heads × Head Dimension × KV Data Type Size

Breaking Down the Formula:

• 2×: Separate storage for Keys and Values
• Batch Size: Number of concurrent requests
• Sequence Length: Maximum context length (input + output)
• Num Layers: Number of transformer blocks
• Num KV Heads: Number of key-value heads (8 for GQA in Qwen3-VL)
• Head Dimension: Size of each attention head (128)
• KV Data Type Size: Bytes per value (typically 2 for float16)

Example Calculation for Qwen3-VL-32B:

Scenario: 1 user, 8,192 token context, float16 KV cache

Batch Size: 1
Sequence Length: 8,192 tokens
Num Layers: 64
Num KV Heads: 8 (Grouped Query Attention)
Head Dimension: 128
KV Data Type: float16 (2 bytes)

KV Cache = 2 × 1 × 8,192 × 64 × 8 × 128 × 2
         = 2 × 1 × 8,192 × 64 × 8 × 256
         = 2 × 1,073,741,824 bytes
         = 2,147,483,648 bytes
         = 2.15 GB (decimal; = 2.00 GiB binary)

Scaling with Batch Size:

Scaling with Sequence Length:

3. Activation Memory

Memory required for intermediate computations during forward passes. The formula below is a planning heuristic, not an exact PyTorch or vLLM allocator formula; attention implementation, prefill chunking, batching and temporary buffers change peak memory.

Illustrative Activation Reserve = Batch Size × Sequence Length × (18 × Hidden Size + 4 × Intermediate Size)

Example Calculation for Qwen3-VL-32B:

Scenario: 1 user, 8,192 token context

Batch Size: 1
Sequence Length: 8,192
Hidden Size: 5,120
Intermediate Size: 25,600

Activation Memory = 1 × 8,192 × (18 × 5,120 + 4 × 25,600)
                  = 8,192 × (92,160 + 102,400)
                  = 8,192 × 194,560
                  = 1,593,835,520 bytes
                  = 1.59 GB (base value)

Data Type Multipliers:

Different quantization levels have different activation memory footprints:

4. Non-PyTorch Memory Overhead

System-level memory overhead for CUDA context, cuBLAS, and other framework components.

Illustrative Runtime Reserve ≈ 1.00 GB

The 1 GB is an illustrative reserve, not a fixed framework cost. The interactive calculator instead uses 1.2 GB scaled by its format-overhead factor (for example ~1.26 GB for int8). Measure the actual runtime peak before choosing a device.

Complete Inference Memory Formula

Combining all components, the total VRAM required for inference:

Total Inference VRAM = (Model Weights + KV Cache + Non-PyTorch Memory + Activations) / GPU Utilization

Capacity reserve: The examples divide by 0.9 to set aside 10% of VRAM. This is a planning choice, not an observed runtime utilization.

Calculator difference: These hand examples use raw payload bytes and an explicit 10% capacity reserve. The interactive calculator adds assumed weight-format overhead — FP8 ×1.03, INT8 ×1.05, INT4/AWQ ×1.12; its NVFP4 entry uses 4.5 effective bits per weight, including block scales. It uses a 1.2 GB runtime reserve multiplied by the format factor and does not divide capacity by the throughput-efficiency slider. The two estimates answer slightly different planning questions; neither guarantees fit.

Example: Qwen3-VL-32B Inference (int8 quantization)

Configuration:

• Quantization: int8 (1 byte per parameter)
• Batch Size: 1 user
• Sequence Length: 8,192 tokens
• KV Cache Data Type: float16
• GPU Utilization: 0.9

Step-by-Step Calculation:

1. Model Weights:
   33,357,390,064 × 1 byte = 33,357,390,064 bytes = 33.36 GB

2. KV Cache (float16):
   2 × 1 × 8,192 × 64 × 8 × 128 × 2 = 2,147,483,648 bytes = 2.15 GB

3. Activations (int8 uses 1.0× multiplier):
   1 × 8,192 × (18 × 5,120 + 4 × 25,600) × 1.0 = 1,593,835,520 bytes = 1.59 GB

4. Non-PyTorch Memory:
   ≈ 1.00 GB

5. Total (before GPU utilization adjustment):
   33.36 + 2.15 + 1.59 + 1.00 = 38.10 GB

6. Adjusted for GPU Utilization (90%):
   38.10 / 0.9 = 42.33 GB

Result: You need approximately 42 GB of VRAM to run Qwen3-VL-32B in int8 quantization with 8k context for a single user.

Example: Qwen3-VL-32B Inference (int4 quantization)

Configuration:

• Quantization: int4 (0.5 bytes per parameter)
• Batch Size: 4 users
• Sequence Length: 8,192 tokens
• KV Cache Data Type: float16
• GPU Utilization: 0.9

Step-by-Step Calculation:

1. Model Weights:
   33,357,390,064 × 0.5 bytes = 16,678,695,032 bytes = 16.68 GB

2. KV Cache (float16, batch=4):
   2 × 4 × 8,192 × 64 × 8 × 128 × 2 = 8,589,934,592 bytes = 8.59 GB

3. Activations (int4 uses 1.0× multiplier, batch=4):
   4 × 8,192 × (18 × 5,120 + 4 × 25,600) × 1.0 = 6,375,342,080 bytes = 6.38 GB

4. Non-PyTorch Memory:
   ≈ 1.00 GB

5. Total (before GPU utilization adjustment):
   16.68 + 8.59 + 6.38 + 1.00 = 32.65 GB

6. Adjusted for GPU Utilization (90%):
   32.65 / 0.9 = 36.28 GB

Result: You need approximately 36 GB of VRAM to run Qwen3-VL-32B in int4 quantization with 8k context for 4 concurrent users.

Training Memory Requirements

Full-parameter training has a different memory budget from inference. There is no persistent inference KV cache to add: automatic differentiation keeps or recomputes forward activations for the backward pass. Activation memory depends on the attention kernel, checkpointing, microbatch size and sequence length.

For 33,357,390,064 trainable parameters, a simple mixed-precision Adam accounting is:

These are model-state totals across all devices, before activations, temporary buffers, runtime allocations and any safety margin. Optimizer implementation, parameter sharding and offload change where that state resides. Quantized LoRA/QLoRA training freezes the base weights and trains small adapters, so it requires a separate calculation; see LoRA/QLoRA memory math. A device count cannot be obtained by simply dividing the model-state total by a card's VRAM: the chosen parallelism determines how weights, optimizer state and activations are distributed.

GPU Memory Comparison Table

Here's a comprehensive comparison for Qwen3-VL-32B across different quantization methods:

Configuration: Batch size 1, Sequence length 8,192 tokens, GPU utilization 0.9

Practical GPU Recommendations

Based on the calculations above, here are suitable GPU configurations for Qwen3-VL-32B:

Inference Deployment

Full training requires a separate system design for optimizer states, sharding, activations and checkpointing; the training section above gives the model-state subtotal.

Key Takeaways

1. Model Weights Scale Linearly: Doubling parameters doubles weight memory
2. KV Cache Scales with Context: KV cache memory grows linearly with context length
3. Batch Size Multiplies KV Cache: Each concurrent user adds KV cache overhead
4. Quantization Dramatically Reduces Memory: int4 uses ~1/8th the memory of float32
5. Training memory is method-dependent: Full training, LoRA and QLoRA have different trainable states and activation footprints.
6. Leave capacity headroom: Choose a reserve for the actual runtime and verify peak allocation on hardware.

Advanced Considerations

Grouped Query Attention (GQA) Impact

Qwen3-VL-32B uses Grouped Query Attention with 8 KV heads instead of 64 query heads. This reduces KV cache memory by 8× compared to standard Multi-Head Attention:

Standard MHA: 64 KV heads → KV Cache = 17.18 GB (8k context, float16)
GQA (Qwen3): 8 KV heads → KV Cache = 2.15 GB (8k context, float16)

Memory Savings: 15.03 GB (87.5% reduction)

Long Context Scaling

Qwen3-VL-32B supports up to 262k tokens. Here's how KV cache scales:

KV Cache Quantization

Using fp8 for KV cache can halve memory usage:

float16 KV Cache (8k, batch=1): 2.15 GB
fp8 KV Cache (8k, batch=1): 1.07 GB

Savings: 50% memory reduction with minimal quality loss

Conclusion

Accurate VRAM calculation is essential for efficient LLM deployment. By understanding the mathematical foundations:

1. Choose the right quantization for your quality-memory tradeoff
2. Plan for KV cache scaling with concurrent users and context length
3. Budget for training overhead if fine-tuning
4. Select appropriate GPUs based on actual requirements
5. Always include safety margins to prevent OOM errors

Use these formulas to estimate memory requirements for any Hugging Face model by extracting configuration parameters and applying the calculations shown in this guide.

Related

• LLM Inference Math: From Theory to Hardware — the compute side of the same roofline.
• LLM Inference Calculator & Break-Even Analyzer — run the numbers for your model and card.

References and Tools

• Model Configuration: Qwen3-VL-32B-Instruct config.json
• Hugging Face Hub: Model metadata and parameter counts available via API
• GPU Specifications: Check manufacturer specifications for exact VRAM capacities
• Quantization Methods: GPTQ, AWQ, GGUF for different precision levels