LoRA and QLoRA Fine-Tuning: The Actual Memory Math

"Fine-tune a 32B model on one GPU" threads take the inference number and quietly subtract nothing. Reality: training keeps gradients and optimizer state for every trainable parameter, and under mixed-precision AdamW that budget is 16 bytes per parameter — eight times the 2 bytes a bf16 weight costs at inference. This guide derives that budget term by term, then walks the two standard escapes (LoRA's rank decomposition, QLoRA's 4-bit base) with worked numbers for a real model: Qwen3-32B, 32,762,123,264 parameters (count verified against the Hugging Face safetensors metadata of Qwen/Qwen3-32B; config: 64 layers, hidden size 5,120, 64 attention heads / 8 KV heads, head dim 128, FFN 25,600, untied 151,936-token embeddings) ¹. At inference this model needs 65.5 GB just for bf16 weights. This guide ties into our VRAM calculator guide and the quantization guide; the serving-side counterpart is the KV cache guide.

Why training is not inference

The accounting is the ZeRO paper's (arXiv:1910.02054, Sec. 3.1) ², mixed-precision training keeps a 2-byte fp16/bf16 parameter copy and a 2-byte gradient copy, while the optimizer holds an fp32 copy of the parameters plus fp32 momentum and variance — 4+4+4 bytes — for a total of 2Ψ + 2Ψ + 12Ψ = 16Ψ bytes for a model with Ψ parameters. That 14-of-16 byte share sits on trainable parameters only: PyTorch's mixed-precision recipe and DeepSpeed's docs both keep gradients and fp32 master/optimizer state solely for parameters with requires_grad=True.

Qwen3-32B, full fine-tune, AdamW mixed precision. Model state = 32,762,123,264 × 16 B = 524.2 GB (488.2 GiB) — before activations, gradients of embeddings buffers, or the CUDA context. The same model serves in 65.5 GB. That 8× factor is the entire story of this guide. (ZeRO/FSDP sharding and offloading can split the 524 GB across GPUs and host RAM — the bytes don't disappear, they move.)

LoRA: freeze the 99%, train a low-rank delta

LoRA (arXiv:2106.09685) freezes the pretrained weights and learns an update as a product of two small matrices — for each adapted weight matrix $W$, the forward pass computes $W x + \frac{\alpha}{r} B A x$, with $A \in \mathbb{R}^{r \times d_{\text{in}}}$, $B \in \mathbb{R}^{d_{\text{out}} \times r}$, $B$ initialized to zero. The trainable delta $\Delta W = BA$ has exactly:

parameters per adapted matrix — rank 8 adds $8 \cdot (5{,}120 + 8{,}192) = 106{,}496$ parameters to a single Qwen3 q_proj, versus 41,943,040 frozen ones (0.25%). Applied to all seven linear matrices per layer (q, k, v, o, gate, up, down), one Qwen3-32B layer carries $r \times 131{,}072$ adapter parameters:

Full fine-tune model state for reference: 524.2 GB (16 B × 32.76B params). The 16 B/param column includes the bf16 adapter weights themselves, gradients, and fp32 m/v/master. It applies only to the trainable 0.2–1.6%. Actual optimizer implementations can omit the master copy or use different state types.

Why the number collapses: gradients and Adam states exist only for trainable parameters. The frozen base contributes its forward weights, while the adapters carry the training state. The LoRA paper reports a 3× GPU-memory reduction for its GPT-3 175B setup ³; that result is specific to its configuration, not a universal reduction factor.

Two caveats. First, the total saving is bounded below by the frozen base: LoRA at r=16 still needs 65.5 GB of bf16 Qwen3-32B weights; a 70B bf16 base alone needs roughly 140 GB and cannot fit on one 80 GB A100/H100. Second, the 16 B/param budget assumes fp32 master weights and Adam states; 8-bit optimizers use different accounting and require separate measurement.

QLoRA: 4-bit base, same optimizer math

QLoRA (arXiv:2305.14314) removes LoRA's floor — the 2 B/param frozen base — by storing the frozen weights in 4-bit NormalFloat (NF4) with blockwise scales. The paper's accounting, blocksize 64:

Without double quantization the blockwise scales cost 32/64 = 0.5 bits/param — for a 65B model that is ~0.4 GB of fp32 constants per weight bit saved. Double quantization (quantize the first-level constants themselves to FP8 with blocksize 256) reduces that to 8/64 + 32/16384 = 0.127 bits/param, "a reduction of 0.373 bits per parameter" — the paper's exact number, ⁴, roughly 3 GB saved on a 65B model. Paged optimizers, QLoRA's third component, do not shrink steady-state memory: they use NVIDIA unified memory to page optimizer state to CPU during the transient spike that gradient checkpointing causes on long sequences.

Qwen3-32B in QLoRA (NF4 + DQ, LoRA r=16 all-linear, batch 4 × 2,048 tokens, checkpointing on):

• Base weights: 32,762,123,264 × 4.127 bits / 8 = 16.9 GB (vs. 65.5 GB bf16 — a 3.9× cut on the frozen base)
• Adapter trainable state: 134,217,728 × 16 B = 2.15 GB (identical budget to LoRA — QLoRA quantizes only the frozen part)
• Base plus adapter model state: ≈ 19.0 GB, before activations, dequantization buffers, CUDA workspace and allocator overhead. A 24 GB card is therefore not guaranteed to fit the stated batch; measure the actual training peak.

Paged 8-bit AdamW can reduce the optimizer-state part of this budget, but its exact memory use depends on the implementation and the number and size of adapted tensors.

Activation memory and the checkpointing trade

Model state is the floor; activations scale with tokens. Korthikanti et al. derive approximately s·b·h·(34 + 5a·s/h) bytes per layer for the specific transformer implementation they analyze ⁵. Fused attention can avoid storing the quadratic attention matrix, but the remaining 34·s·b·h is an architecture-dependent illustration, not an exact Qwen3 allocation. Gradient checkpointing saves region inputs instead of all intermediates and recomputes them during backward ⁶. Crucially, the saved boundary tensors are not the peak: temporary activations for the region being recomputed, gradients, and workspaces remain live.

For a 64-layer, 5,120-hidden model at batch 4 × 2,048, merely retaining one bf16 layer-boundary tensor per layer costs 64 × 8,192 × 5,120 × 2 ≈ 5.37 GB before the backward peak. The former 0.67 GB checkpoint estimate counted only eight boundaries and ignored recomputation within an eight-layer segment; it cannot be used as a GPU-fit figure. Full-layer recomputation can approach one extra forward pass, roughly +33% if backward costs twice forward, but actual overhead depends on checkpoint granularity and kernels ⁷.

Illustrative 64-layer activation accounting (h = 5,120, bf16; the 34 coefficient comes from the cited transformer implementation, not a Qwen3 measurement):

The right column is only a boundary-tensor count. It does not upper-bound peak memory under checkpointing. For a real fit decision, include the recomputation peak and use torch.cuda.max_memory_allocated() on the intended model, sequence length, batch, optimizer, and training stack.

Model-state floors by method

These are weight, gradient, and optimizer-state floors. They exclude activations and temporary buffers, so they do not establish that a particular GPU configuration will fit:

Llama-3.1-8B parameter count is from its Hugging Face metadata (meta-llama/Llama-3.1-8B: 8,030,261,248). The 70B row is rounded; actual checkpoint counts and adapter shapes differ. QLoRA's paper demonstrated a 65B model on a 48 GB GPU under its own training configuration ⁴, which does not establish a fit for every 70B model or batch.

What fine-tuning does and does not guarantee

Memory savings do not establish task quality. Evaluate the adapted model on the facts and behaviours that matter:

• Factual updates need evaluation. Fine-tuning can change factual responses, but neither LoRA's low-rank analysis nor QLoRA's instruction-data results prove that a small SFT dataset reliably installs or maintains a body of changing facts ^{9 10}. Retrieval can supply current source material; verify both retrieval and generated answers.

• Capacity does not price throughput. At batch 4 × 2,048 tokens, a 100M-token one-epoch dataset requires at least 12,208 optimizer steps if every token contributes and no examples are discarded. Checkpointing and dequantization add work. Measure tokens per second on the target stack before estimating run time or cost.

• No honest defaults without tuning ¹¹. The QLoRA paper's own hyperparameter table uses learning rate 2e-4 for 7B and 13B models and 1e-4 for 33B and up, batch size 16, constant schedule — roughly 10x higher than full fine-tuning rates, because only the small adapter population is being optimized and the base is frozen. LoRA's paper likewise sweeps LR per task. Any "just use" number — rank 8, epochs 3, LR 2e-4 — is a starting grid, not a result; the honest approach is a small sweep of learning rate and rank before believing a loss curve.

• Quantization error is workload-dependent ¹⁰. QLoRA reports strong NF4 results on its evaluation suite; that does not give a universal quality bound for every model, task, or context length. Benchmark against the intended full-precision baseline.

What to remember

• Training budget per trainable parameter: 16 bytes under mixed-precision AdamW (2 bf16 weight + 2 bf16 grad + 4 fp32 master + 4 m + 4 v), per the ZeRO paper — 8x inference weights. Qwen3-32B full fine-tune: 524.2 GB before activations.
• Frozen parameters retain their forward weights; the 16-byte mixed-precision Adam budget applies to trainable parameters. Qwen3-32B with all-linear LoRA r=16 has about 134M trainable parameters and a 67.7 GB model-state floor.
• LoRA adapter size is r(d_in+d_out) per adapted matrix; the model's seven linears per layer give 67M trainable parameters at r=8 and 537M at r=64.
• QLoRA's frozen NF4 base is about 4.127 bits/parameter under the paper's double-quantization assumptions. Qwen3-32B then has a 16.9 GB base plus 2.15 GB of r=16 adapter state, before activations and workspaces.
• Checkpoint boundaries are not peak activation memory. At batch 4 × 2,048, one bf16 boundary tensor per Qwen3-32B layer already totals about 5.37 GB. Measure the backward peak before claiming a GPU fit.
• Factual quality must be tested after fine-tuning; use current retrieval sources where the application needs current or auditable facts.

For serving-side memory math, see the KV cache guide; for weight formats and 4-bit block quantization mechanics, the quantization guide; for the compute-cost side of running what you trained, the VRAM calculator guide and the inference economics tool.