LLM Quantization: Bit Layouts, Block Scales, and the VRAM Math

Every LLM deployment decision — which GPU, how many concurrent users, whether the model fits at all — starts with one number: bytes per weight. Quantization is the highest-leverage knob you have, and it is worth understanding at the bit level, because the differences between formats are not marketing: they are concrete bit layouts with countable overheads and measurable accuracy costs.

This guide goes lower than most: what the bits inside FP8/FP4 actually encode, how block-scale metadata adds up, and how each popular format (NVFP4, MXFP4, FP8, GPTQ, AWQ, GGUF k-quants) spends its bits. Every number here is sourced from the primary papers, the OCP specification, or vendor documentation — and every size example is computed from the same formulas the site's VRAM calculator uses.

The One Formula That Matters

Model weights in memory:

where $P$ is the parameter count and $b_{\text{eff}}$ is the effective bits per weight — the storage format's payload bits plus all scale/zero-point metadata, divided by the group size. The difference between "4-bit" formats is entirely in $b_{\text{eff}}$: naive 4-bit is 4.0, but real formats land between 4.125 and 4.85 bits per weight. On the Qwen3-VL-32B reference model (~33.4B parameters — 33,357,390,064 per the Hugging Face safetensors census — used throughout our VRAM deep dive), that difference is several gigabytes.

Generalized group overhead — this single expression generates almost every effective-bits number in this article:

$k$ = elements per group/block, $b$ = payload bits per element, $s_{\text{scale}}$ and $s_{\text{zp}}$ = bits for the scale (and optionally a zero-point) shared by the group. A 4-bit payload with one FP16 scale per 128 weights costs $(128 \times 4 + 16)/128 = 4.125$ bits per weight; adding a 16-bit zero-point brings it to 4.25.

How a Float Becomes Bits

A binary floating-point number is three fields: sign $s$, exponent $e$, mantissa $m$. The IEEE-754 pattern, generalized:

The exponent stores a power-of-two window; the mantissa stores position inside that window. Normal numbers have an implicit leading 1 ($1.mmm$); when $e = 0$ the number is subnormal ($0.mmm$), extending the range down to zero at fixed spacing. The bias is set per format so the exponent range is centered usefully.

The 8-bit and 4-bit formats used in LLM inference are all this same structure with fewer bits:

Two asymmetries in that table are deliberate, and both come from the FP8 formats paper that standardized E4M3/E5M2 (arXiv 2209.05433)¹:

• E4M3 has no infinity and exactly one NaN pattern. Freeing those encodings extends its usable range to $448 = 1.75 \times 2^8$. It is the weights/activations format: transformer weight distributions need mantissa precision more than range.
• E5M2 follows IEEE conventions ( infinity + NaN), giving a much wider range ($57{,}344 = 1.75 \times 2^{15}$) at lower precision. It is the gradients format, where magnitude spread dominates.

E2M1: All Sixteen Codes, On the Record

The 4-bit element at the bottom of every FP4 format is E2M1. With one mantissa bit and bias 1, its entire decode table has 16 entries — 8 magnitudes, each signed:

Sign bit set → the negative of each. That is the entire representable universe of FP4: $\pm\{0,\ 0.5,\ 1,\ 1.5,\ 2,\ 3,\ 4,\ 6\}$ — sixteen codes and — because +0 and −0 encode one value — fifteen distinct values, exactly. Raw E2M1 is far too coarse for transformer weights (which is why a 4-bit format without scaling is unusable); the entire low-bit-format design problem is how to reuse these sixteen levels block-by-block so they cover each weight neighborhood well. Everything else in this article is that trick, at different granularity settings.

The Granularity Ladder

Quantization error is dominated by outliers: one large weight in a group forces the scale up and wastes resolution on everything smaller. The historical fix is ever-finer scales:

1. Per-tensor — one scale for all weights. Cheapest ($\approx 0$ overhead) and least accurate. Classic FP8 inference works this way: a single FP32 scale per tensor, applied in software.¹
2. Per-channel — one scale per output row of each weight matrix. Still nearly free, much better. This is the default granularity of GPTQ and AWQ.
3. Per-group (k=128) — one scale per 128 weights. GPTQ/AWQ's "small group size" setting; the AWQ paper notes round-to-nearest is already "quite strong" at group 128 with careful scaling (arXiv 2306.00978).²
4. Per-block, power-of-two scale (k=32, E8M0) — the OCP MX standard: 32 E2M1 elements share one 8-bit power-of-two exponent (arXiv-tracked OCP MX v1.0 spec; the shared scale multiplies by a pure $2^n$, so dequantization is an exponent add).³
5. Per-block, FP8 scale + second level (k=16, E4M3 + FP32) — NVFP4, Blackwell's native format, detailed below.

Each rung adds scale bits — that is the $b_{\text{eff}}$ arithmetic from the first section — and each rung contains outliers more tightly. The jump from rung 1 to rung 5 is roughly a 100× finer scaling mesh for about 0.5 bits per weight of metadata.

NVFP4 vs MXFP4: The Same Payload, Different Rulers

Two 4-bit block-floating formats ship on current hardware, and their difference is a textbook lesson in scale design. Both store E2M1 elements; neither touches the payload bits. Only the ruler differs:

Both layouts come straight from the NVIDIA NVFP4 introduction and the OCP MX specification.⁴³ NVFP4's two changes attack the two error sources:

• Half-size blocks (16 vs 32): one outlier contaminates half as many neighbors.
• A fractional scale (E4M3 vs E8M0): an E8M0 scale must snap to the nearest power of two, so a block whose values sit between $2^n$ and $2^{n+1}$ loses up to half its representable range at the top. An E4M3 scale can land anywhere in the FP8 dynamic range. NVIDIA measures ~88% lower quantization error for E4M3 micro-block scaling versus power-of-two scaling, and in a head-to-head pretraining comparison MXFP4 needed ~36% more tokens to reach the same loss as NVFP4.⁵

The cost of the finer ruler is countable: 4.50 vs 4.25 bits per element, plus one FP32 per tensor (negligible at scale). And because the E4M3 scale has a narrower range than E8M0, NVFP4 adds the per-tensor FP32 second level to re-normalize whole tensors so every block's E4M3 scale stays well-conditioned — global range up top, local fit per block.

Accuracy, from NVIDIA's own PTQ study on DeepSeek-R1-0528 (converted FP8 → NVFP4): within 1% of FP8 across seven language-modeling evaluations, with AIME 2024 actually 2 points higher (89 → 91); MMLU-Pro 85 → 84, GPQA 81 → 80.⁴⁵ The punchline — a 4-bit footprint behaving like an 8-bit model for most serving purposes:

which is exactly how NVIDIA states it (~3.5× and ~1.8×).⁴ On Blackwell tensor cores, FP4 dense throughput is 2× the FP8 rate — the same generational doubling FP8 did to FP16 on Hopper.

Weight-Only Integer Formats: GPTQ and AWQ

GPU hardware before Blackwell has no FP16×INT4 matrix path, so 4-bit integer formats are weight-only formats (W4A16): weights stored in 4 bits, dequantized on the fly, activations kept in FP16. The GPTQ paper is explicit that its speedups come from reduced memory movement, not cheaper math — there is no mixed-precision multiply on mainstream architectures (arXiv 2210.17323).⁶ For decode-time LLM inference that is precisely what you want: one user's token generation is memory-bandwidth-bound (see our inference math guide), so a 4× smaller weight stream is a near-4× faster token stream until you hit other ceilings.

The two families differ in how they choose the rounding:

• GPTQ (arXiv 2210.17323) quantizes weights column-by-column, propagating the rounding error of each column into the not-yet-quantized ones using an approximate second-order (Hessian-based) correction. It compensates errors, needs a calibration set, and quantizes OPT-175B-class models in hours, not days.⁶
• AWQ (arXiv 2306.00978, MLSys 2024 Best Paper) observes that 0.1–1% of weight channels are salient — identifiable from the activation distribution, not the weights — and protects them by searching a per-channel scaling that makes the full-4-bit representation of salient channels more accurate. No backprop, no reconstruction: it generalizes across domains and modalities without overfitting the calibration set.²

Both typically ship at group size 128 with FP16 scales: $b_{\text{eff}} = (128 \times 4 + 16)/128 = 4.125$ bits/weight before zero-points and packing metadata. The interactive calculator models this class of format with a flat 12% overhead factor (int4/AWQ ×1.12) to stay conservative across implementations; hand-check with the formula above when it matters.

One warning from the low level: the scale layout — how scales are interleaved with packed values in memory — varies between exports (block-interleaved, scales-after-values, per-K, tile-major) and is a recurring source of serving-stack bugs when a checkpoint quantized by tool A is loaded by runtime B. If a 4-bit model produces garbage, suspect the scale layout before the model.

GGUF K-Quants: The Byte Layouts, Verified

llama.cpp's k-quants are the most deployed weight quantizers in the world, and they document their own arithmetic. The 2023-of-record k-quant PR defines the super-block structure (QK_K = 256), and llama.cpp's source carries static_asserts that force the byte counts; these are the published per-weight bit costs:⁷

Read Q4_K's row as the generic formula again: 2 bytes d + 2 bytes dmin + 12 bytes packed 6-bit scales + $(256 \times 4)/8 = 128$ bytes of nibbles = 144 bytes per 256 weights — $144 \times 8 / 256 = 4.5$ bits per weight, plus sixteen levels with a two-level scale hierarchy (super-block FP16 scale over per-block 6-bit scales) so fine scaling doesn't cost a full FP16 per 32 weights.

The _S/_M/_L suffixes are a tensor policy, not a format: they pick which tensors get stored above base width (Q4_K_M keeps attention.v and MLP down-proj at higher precision than Q4_K_S). The published effective bpw therefore varies slightly per model: Q4_K_M measures 4.84 bpw on Llama-2-7B, 4.83 on 13B, 4.80 on 70B — the llama.cpp quantize README's own table.⁷ Quality ordering from llama.cpp's published perplexity runs (Llama-2-7B): Q8_0 at 5.9070 vs F16 5.9066 — effectively indistinguishable; Q4_0/K variants sit between; and quality-wise the community default Q4_K_M costs roughly 1% perplexity for ~30% of the FP16 size.

Putting the Formats Side by Side

Effective bits and what you pay for them, using $b_{\text{eff}}$ from the group-overhead formula (integer formats at typical group 128; NVFP4 at 4.50; GGUF at published bpw):

Three honest observations fall out of the table:

1. "4-bit" spans a ~16% size range (17.7–20.6 GB) purely from scale metadata. When a quantized model "doesn't quite fit", the metadata is where the budget went.
2. FP8 has effectively won the high end. Flagship open models ship FP8 checkpoints first (Qwen3-Next, DeepSeek), and E4M3 is the default inference format on Hopper and Blackwell. 16→8 bit halves the footprint at ≈zero serving-quality cost.
3. The 4-bit formats are now quality-competitive with 8-bit. NVFP4's ≤1% delta vs FP8 and Q6_K's near-nil cost mean the 4-5 bit band delivers 3.2–3.8× more model per GB than FP16 — with quantization error now dominated by calibration quality, not bit count.

Which Format Should You Run?

Self-hosted, single-user or small-batch (llama.cpp/LM Studio/Ollama): start Q4_K_M. Size any step-up with $P \times b_{\text{eff}}/8$ against your VRAM after reserving KV-cache (the break-even calculator's KV math applies unchanged). If code/math quality matters, Q6_K; if you are establishing a reference, Q8_0.

vLLM/SGLang/TensorRT-LLM serving on Hopper: FP8 (E4M3 weights, per-tensor or per-block scales per your runtime's support). 4-bit integer weights (AWQ/GPTQ) when VRAM pressure demands it — expect a measurable but small quality step.

Blackwell serving at scale: NVFP4. It is the native format of the hardware's 4-bit tensor cores, the ecosystem (vLLM/TRT-LLM model optimization) supports it directly, and the accuracy tax vs FP8 is ~1% per NVIDIA's measurements. MXFP4 remains the OCP-standard option with broader cross-vendor portability (AMD's MI-series supports MX-family formats).

Whatever you pick, measure. Publish your own before/after on your workload — the perplexity deltas above are Llama-family measurements under llama.cpp calibration; arrival distributions, long-context regimes, and tool-calling patterns shift them. The formulas in this article give you the frontier; only your benchmark tells you where on it to sit.

Further Reading

• LLM VRAM Requirements: A Mathematical Deep Dive — the full weight + KV-cache memory model this article's examples plug into.
• LLM Inference Math: From Theory to Hardware — why decode is memory-bound, which is why $b_{\text{eff}}$ is also your throughput knob.
• How LLMs Actually Work — tokenizers, attention and sampling background for the format decisions above.