LoRA and QLoRA Fine-Tuning: The Actual Memory Math

"Fine-tune a 32B model on one GPU" threads take the inference number and quietly subtract nothing. Reality: training keeps gradients and optimizer state for every trainable parameter, and under mixed-precision AdamW that budget is 16 bytes per parameter — eight times the 2 bytes a bf16 weight costs at inference. This guide derives that budget term by term, then walks the two standard escapes (LoRA's rank decomposition, QLoRA's 4-bit base) with worked numbers for a real model: Qwen3-32B, 32,762,123,264 parameters (count verified against the Hugging Face safetensors metadata of Qwen/Qwen3-32B; config: 64 layers, hidden size 5,120, 64 attention heads / 8 KV heads, head dim 128, FFN 25,600, untied 151,936-token embeddings) ¹. At inference this model needs 65.5 GB just for bf16 weights. This guide ties into our VRAM calculator guide and the quantization guide; the serving-side counterpart is the KV cache guide.

Why training is not inference

The accounting is the ZeRO paper's (arXiv:1910.02054, Sec. 3.1) ², mixed-precision training keeps a 2-byte fp16/bf16 parameter copy and a 2-byte gradient copy, while the optimizer holds an fp32 copy of the parameters plus fp32 momentum and variance — 4+4+4 bytes — for a total of 2Ψ + 2Ψ + 12Ψ = 16Ψ bytes for a model with Ψ parameters. That 14-of-16 byte share sits on trainable parameters only: PyTorch's mixed-precision recipe and DeepSpeed's docs both keep gradients and fp32 master/optimizer state solely for parameters with requires_grad=True.

Qwen3-32B, full fine-tune, AdamW mixed precision. Model state = 32,762,123,264 × 16 B = 524.2 GB (488.2 GiB) — before activations, gradients of embeddings buffers, or the CUDA context. The same model serves in 65.5 GB. That 8× factor is the entire story of this guide. (ZeRO/FSDP sharding and offloading can split the 524 GB across GPUs and host RAM — the bytes don't disappear, they move.)

LoRA: freeze the 99%, train a low-rank delta

LoRA (arXiv:2106.09685) freezes the pretrained weights and learns an update as a product of two small matrices — for each adapted weight matrix $W$, the forward pass computes $W x + \frac{\alpha}{r} B A x$, with $A \in \mathbb{R}^{r \times d_{\text{in}}}$, $B \in \mathbb{R}^{d_{\text{out}} \times r}$, $B$ initialized to zero. The trainable delta $\Delta W = BA$ has exactly:

parameters per adapted matrix — rank 8 adds $2 \cdot 8 \cdot (5{,}120 + 8{,}192) = 212{,}992$ parameters to a single Qwen3 q_proj, versus 41,943,040 frozen ones (0.51%). Applied to all seven linear matrices per layer (q, k, v, o, gate, up, down), one Qwen3-32B layer carries $2r \times 131{,}072$ adapter parameters:

Full fine-tune model state for reference: 524.2 GB (16 B × 32.76B params). "Optimizer+grad state" includes the bf16 adapter weights themselves plus their bf16 gradients and fp32 m/v/master — i.e. the same 16 B/param budget from Section 1, applied only to the trainable 0.4–3.3%.

Why the number collapses so hard: gradients, fp32 master copies, and Adam's m/v exist only for trainable parameters. Freeze 99.2% of the model and 99.2% of that 524 GB evaporates — the frozen base contributes only its 2 B/param forward weights. The LoRA paper's GPT-3 175B numbers ³ are the same effect at larger scale: 4.7M trainable weights against 175B frozen, 350 GB of VRAM versus about 1.2 TB, and a checkpoint of 35 MB versus 350 GB — a 10,000x cut in trainable parameters and (as the paper's abstract states) a 3x cut against full Adam fine-tuning.

Two honest caveats. First, the total saving is bounded below by the frozen base: LoRA at r=16 still needs the full 65.5 GB of bf16 weights, so at 70B scale LoRA buys you "fits on one A100/H100 80 GB", not "fits on your laptop". Second, the 16 B/param itself assumes full fp32 Adam state; 8-bit Adam variants (bitsandbytes) shave the 12 optimizer bytes down to ~4, mostly relevant when the trainable fraction is large.

QLoRA: 4-bit base, same optimizer math

QLoRA (arXiv:2305.14314) removes LoRA's floor — the 2 B/param frozen base — by storing the frozen weights in 4-bit NormalFloat (NF4) with blockwise scales. The paper's accounting, blocksize 64:

Without double quantization the blockwise scales cost 32/64 = 0.5 bits/param — for a 65B model that is ~0.4 GB of fp32 constants per weight bit saved. Double quantization (quantize the first-level constants themselves to FP8 with blocksize 256) reduces that to 8/64 + 32/16384 = 0.127 bits/param, "a reduction of 0.373 bits per parameter" — the paper's exact number, ⁴, roughly 3 GB saved on a 65B model. Paged optimizers, QLoRA's third component, do not shrink steady-state memory: they use NVIDIA unified memory to page optimizer state to CPU during the transient spike that gradient checkpointing causes on long sequences.

Qwen3-32B in QLoRA (NF4 + DQ, LoRA r=16 all-linear, batch 4 × 2,048 tokens, checkpointing on):

• Base weights: 32,762,123,264 × 4.127 bits / 8 = 16.9 GB (vs. 65.5 GB bf16 — a 3.9× cut on the frozen base)
• Adapter trainable state: 268,435,456 × 16 B = 4.0 GB (identical budget to LoRA — QLoRA quantizes only the frozen part)
• Activations, checkpointed: ≈ 0.7 GB (next section)
• Peak model-state total: ≈ 21.9 GB — a 24 GB card (RTX 3090/4090-class) trains a 32B model. The same card holds Qwen3-32B inference weights (65.5 GB) only after quantizing to ~4 bits.

With paged 8-bit AdamW (bitsandbytes paged_adamw_8bit), the adapter state drops from 4.0 GB toward 4.0 × (6/16) ≈ 1.5 GB — small here, decisive at r=256 on a 70B model.

Activation memory and the checkpointing trade

Model state is the floor; activations scale with tokens, not parameters. Following the activation-memory derivation of Korthikanti et al. (arXiv:2205.05198, Megatron-LM's selective recompute paper) ⁵, a transformer layer without attention-matrix materialization (FlashAttention) needs ≈ 34·s·b·h bytes per layer at fp16/bf16 — attention block 11sbh plus 19sbh for the MLP block in the paper's Section 4.1 accounting (Eq. 1: sbh(34 + 5as/h) per layer), plus the s²b terms that FlashAttention removes. With gradient checkpointing at the classic √L spacing (one checkpoint every √64 = 8 layers), peak stored activation memory drops from 34·s·b·h·L to about 2·√L·s·b·h — a 136× cut for this model (34·64 = 2,176 vs 2·8 = 16 per-unit coefficients), at the cost of one extra forward per segment.

The exchange rate is one extra forward pass per step: a normal step is forward (1 unit) + backward (2 units) = 3; a checkpointed step is 4 units — +33% in the unit model (3 units of work per normal step, 4 with one extra forward), the framing both Chen et al. (arXiv:1604.06174) ⁶ and the PyTorch activation-checkpointing docs use — Chen's measured overhead on their benchmarks is ~30%. Memorable as "pay a third, quarter your activation memory."

Qwen3-32B activation table (h = 5,120, L = 64, bf16, FlashAttention):

Read the ratio, not just the magnitudes: activations are linear in tokens and irrelevant at low batch, but at batch 8 × 8k context they outgrow the entire 21.9 GB QLoRA model state 33-fold without checkpointing — and stay under 6 GB with it. (Real peaks include fragmentation and cudnn workspaces; treat these as lower bounds and check torch.cuda.max_memory_allocated().)

What fits where: model × method → GPU

Combining the three budgets (PEFT optimizer state + frozen-base bytes + activation span at batch 4 × 2,048, checkpointed):

Llama-3.1-8B parameter count verified against Hugging Face metadata (meta-llama/Llama-3.1-8B: 8,030,261,248). The 70B row uses the paper-class Llama-2-70B figure of 68.96B effective parameters for adapter math — QLoRA's own headline result is the third column made famous: "finetune a 65B parameter model on a single 48GB GPU."

The critical view: what fine-tuning will not do

Memory math says fine-tuning is cheap; the hype reads that as powerful. Both papers, read carefully, say the opposite of the marketing:

• Fine-tuning teaches form, not facts ⁸. LoRA's own analysis found its updates live in a low-rank subspace, and the QLoRA paper's data-mixing experiments show adapter quality tracking the instruction data quality, not quantity — a model that lacks knowledge cannot acquire it from a 1,000-example SFT run; it learns style and format. Use retrieval for facts, not adapters.

• "Cheap" counts only VRAM, never throughput ⁹. Yes, QLoRA fits a 32B model in 22 GB. At batch 4 × 2,048 tokens, one training step processes 8,192 tokens — a 100M-token SFT dataset takes 12,207 steps, and each step carries the +33% checkpointing tax plus 4-bit dequantization overhead. QLoRA's Guanaco 65B needed 24 hours on a professional GPU for a single run; a memory-feasible setup is not a time-feasible setup, and "fine-tune your own model for the price of API credits" quietly skips that the same dataset, LoRA-rank tuning, retries and evaluation multiply the bill.

• No honest defaults without tuning ¹⁰. The QLoRA paper's own hyperparameter table uses learning rate 2e-4 for 7B and 13B models and 1e-4 for 33B and up, batch size 16, constant schedule — roughly 10x higher than full fine-tuning rates, because only the small adapter population is being optimized and the base is frozen. LoRA's paper likewise sweeps LR per task. Any "just use" number — rank 8, epochs 3, LR 2e-4 — is a starting grid, not a result; the honest approach is a small sweep of learning rate and rank before believing a loss curve.

• The quantization error is real but bounded ⁹. QLoRA's NF4 claim is that 4-bit quantized fine-tuning matches 16-bit fine-tuning quality for the adapter learning problem — validated on its benchmarks, not a law of nature. QLoRA's ablations show NF4 specifically is required (other 4-bit types degrade), and both papers agree the frozen base's knowledge and the adapter's data chemistry dominate the outcome more than any memory-side trick.

What to remember

• Training budget per trainable parameter: 16 bytes under mixed-precision AdamW (2 bf16 weight + 2 bf16 grad + 4 fp32 master + 4 m + 4 v), per the ZeRO paper — 8x inference weights. Qwen3-32B full fine-tune: 524.2 GB before activations.
• Frozen parameters cost 2 bytes, not 16. That asymmetry, not any clever matrix trick, is LoRA's whole saving: r=16 all-linear on Qwen3-32B trains 0.82% of the model → 69.5 GB model state, 13.3% of full fine-tuning.
• LoRA adapter size is 2r(d_in+d_out) per matrix, all seven linears per layer → 268M trainable at r=16 on the worked model (scales linearly in r: 134M at r=8, 1.07B at r=64).
• QLoRA's frozen base costs 4.127 bits/param (NF4 4 + FP8 scales 0.125 + second-level 0.002), with double quantization saving exactly 0.373 bits/param vs. blockwise fp32 scales — Qwen3-32B: 16.9 GB base, ≈22 GB total peak at r=16, batch 4 × 2k, checkpointed.
• Activations scale with tokens: ≈ 34·s·b·h bytes per layer, √L-checkpointing cuts L→√L for +33% compute (one extra forward pass). At batch 8 × 8k context, activations are 730 GB uncheckpointed, 5.4 GB checkpointed — often the real OOM, not the model.
• Fine-tuning transfers style and format, injects knowledge poorly — both papers' data sections support limits, use RAG for facts, and count throughput, not just VRAM, before promising a "cheap custom model."

For serving-side memory math, see the KV cache guide; for weight formats and 4-bit block quantization mechanics, the quantization guide; for the compute-cost side of running what you trained, the VRAM calculator guide and the inference economics tool.