LLM Quantization: Bit Layouts, Block Scales, and the VRAM Math

Start with a number you can move: Compare INT4 and NVFP4 visually to see rounding, scaling and block overhead before the bit-level reference below.

Every LLM deployment decision — which GPU, how many concurrent users, whether the model fits at all — starts with one number: bytes per weight. Quantization is the highest-leverage knob you have, and it is worth understanding at the bit level, because the differences between formats are not marketing: they are concrete bit layouts with countable overheads and measurable accuracy costs.

This guide goes lower than most: what the bits inside FP8/FP4 actually encode, how block-scale metadata adds up, and how each popular format (NVFP4, MXFP4, FP8, GPTQ, AWQ, GGUF k-quants) spends its bits. Every number here is sourced from the primary papers, the OCP specification, or vendor documentation — and every size example is computed from the same formulas the site's VRAM calculator uses.

The One Formula That Matters

Model weights in memory:

where $P$ is the parameter count and $b_{\text{eff}}$ is the effective bits per weight — payload bits plus amortized scale/zero-point metadata. Real 4-bit formats have different effective widths, and mixed-precision tensors can change the checkpoint-wide average further. On the Qwen3-VL-32B reference model (~33.4B parameters — 33,357,390,064 per the Hugging Face safetensors census — used throughout our VRAM deep dive), these differences can amount to gigabytes.

Generalized group overhead — this single expression generates almost every effective-bits number in this article:

$k$ = elements per group/block, $b$ = payload bits per element, $s_{\text{scale}}$ and $s_{\text{zp}}$ = bits for the scale (and optionally a zero-point) shared by the group. A 4-bit payload with one FP16 scale per 128 weights costs $(128 \times 4 + 16)/128 = 4.125$ bits per weight; adding a 16-bit zero-point brings it to 4.25.

How a Float Becomes Bits

A binary floating-point number is three fields: sign $s$, exponent $e$, mantissa $m$. The IEEE-754 pattern, generalized:

The exponent stores a power-of-two window; the mantissa stores position inside that window. Normal numbers have an implicit leading 1 ($1.mmm$); when $e = 0$ the number is subnormal ($0.mmm$), extending the range down to zero at fixed spacing. The bias is set per format so the exponent range is centered usefully.

The 8-bit and 4-bit formats used in LLM inference are all this same structure with fewer bits:

Two asymmetries in that table are deliberate, and both come from the FP8 formats paper that standardized E4M3/E5M2 (arXiv 2209.05433)¹:

• E4M3 has no infinity and exactly one NaN pattern. Freeing those encodings extends its usable range to $448 = 1.75 \times 2^8$. It is the weights/activations format: transformer weight distributions need mantissa precision more than range.
• E5M2 follows IEEE conventions ( infinity + NaN), giving a much wider range ($57{,}344 = 1.75 \times 2^{15}$) at lower precision. It is the gradients format, where magnitude spread dominates.

E2M1: All Sixteen Codes, On the Record

The 4-bit element at the bottom of every FP4 format is E2M1. With one mantissa bit and bias 1, its entire decode table has 16 entries — 8 magnitudes, each signed:

Sign bit set → the negative of each. That is the entire representable universe of FP4: $\pm\{0,\ 0.5,\ 1,\ 1.5,\ 2,\ 3,\ 4,\ 6\}$ — sixteen codes and — because +0 and −0 encode one value — fifteen distinct values, exactly. Raw E2M1 is far too coarse for transformer weights (which is why a 4-bit format without scaling is unusable); the entire low-bit-format design problem is how to reuse these sixteen levels block-by-block so they cover each weight neighborhood well. Everything else in this article is that trick, at different granularity settings.

The Granularity Ladder

Quantization error is dominated by outliers: one large weight in a group forces the scale up and wastes resolution on everything smaller. The historical fix is ever-finer scales:

1. Per-tensor — one scale for all weights. Cheapest ($\approx 0$ overhead) and least accurate. Classic FP8 inference works this way: a single FP32 scale per tensor, applied in software.¹
2. Per-channel — one scale per output row of a weight matrix. It is one possible weight-quantization granularity, with low metadata cost.
3. Per-group (k=128) — one scale per 128 weights. A common GPTQ/AWQ checkpoint setting; the AWQ paper notes round-to-nearest is already "quite strong" at group 128 with careful scaling (arXiv 2306.00978).²
4. Per-block, power-of-two scale (k=32, E8M0) — the OCP MX standard: 32 E2M1 elements share one 8-bit power-of-two exponent (arXiv-tracked OCP MX v1.0 spec; the shared scale multiplies by a pure $2^n$, so dequantization is an exponent add).³
5. Per-block, FP8 scale + second level (k=16, E4M3 + FP32) — NVFP4, Blackwell's native format, detailed below.

Each rung adds scale bits — that is the $b_{\text{eff}}$ arithmetic from the first section — and each rung contains outliers more tightly. The jump from rung 1 to rung 5 is roughly a 100× finer scaling mesh for about 0.5 bits per weight of metadata.

NVFP4 vs MXFP4: The Same Payload, Different Rulers

Two 4-bit block-floating formats ship on current hardware, and their difference is a textbook lesson in scale design. Both store E2M1 elements; neither touches the payload bits. Only the ruler differs:

Both layouts come straight from the NVIDIA NVFP4 introduction and the OCP MX specification.⁴³ NVFP4's two changes attack the two error sources:

• Half-size blocks (16 vs 32): one outlier contaminates half as many neighbors.
• A fractional scale (E4M3 vs E8M0): an E8M0 scale must snap to the nearest power of two, so a block whose values sit between $2^n$ and $2^{n+1}$ loses up to half its representable range at the top. An E4M3 scale can land anywhere in the FP8 dynamic range. NVIDIA measures ~88% lower quantization error for E4M3 micro-block scaling versus power-of-two scaling, and in a head-to-head pretraining comparison MXFP4 needed ~36% more tokens to reach the same loss as NVFP4.⁵

The cost of the finer ruler is countable: 4.50 vs 4.25 bits per element, plus one FP32 per tensor (negligible at scale). And because the E4M3 scale has a narrower range than E8M0, NVFP4 adds the per-tensor FP32 second level to re-normalize whole tensors so every block's E4M3 scale stays well-conditioned — global range up top, local fit per block.

Accuracy, from NVIDIA's own PTQ study on DeepSeek-R1-0528 (converted FP8 → NVFP4): within 1% of FP8 across seven language-modeling evaluations, with AIME 2024 actually 2 points higher (89 → 91); MMLU-Pro 85 → 84, GPQA 81 → 80.⁴⁵ The punchline — a 4-bit footprint behaving like an 8-bit model for most serving purposes:

which is exactly how NVIDIA states it (~3.5× and ~1.8×).⁴ On Blackwell tensor cores, FP4 dense throughput is 2× the FP8 rate — the same generational doubling FP8 did to FP16 on Hopper.

Weight-Only Integer Formats: GPTQ and AWQ

GPU hardware before Blackwell has no FP16×INT4 matrix path, so 4-bit integer formats are weight-only formats (W4A16): weights stored in 4 bits, dequantized on the fly, activations kept in FP16. The GPTQ paper is explicit that its speedups come from reduced memory movement, not cheaper math — there is no mixed-precision multiply on mainstream architectures (arXiv 2210.17323).⁶ For decode-time LLM inference that is precisely what you want: one user's token generation is memory-bandwidth-bound (see our inference math guide), so a 4× smaller weight stream is a near-4× faster token stream until you hit other ceilings.

The two families differ in how they choose the rounding:

• GPTQ (arXiv 2210.17323) quantizes weights column-by-column, propagating the rounding error of each column into the not-yet-quantized ones using an approximate second-order (Hessian-based) correction. It compensates errors, needs a calibration set, and quantizes OPT-175B-class models in hours, not days.⁶
• AWQ (arXiv 2306.00978, MLSys 2024 Best Paper) observes that 0.1–1% of weight channels are salient — identifiable from the activation distribution, not the weights — and protects them by searching a per-channel scaling that makes the full-4-bit representation of salient channels more accurate. No backprop, no reconstruction: it generalizes across domains and modalities without overfitting the calibration set.²

Both typically ship at group size 128 with FP16 scales: $b_{\text{eff}} = (128 \times 4 + 16)/128 = 4.125$ bits/weight before zero-points and packing metadata. The interactive calculator models this class of format with a flat 12% overhead factor (int4/AWQ ×1.12) to stay conservative across implementations; hand-check with the formula above when it matters.

One warning from the low level: the scale layout — how scales are interleaved with packed values in memory — varies between exports (block-interleaved, scales-after-values, per-K, tile-major) and is a recurring source of serving-stack bugs when a checkpoint quantized by tool A is loaded by runtime B. If a 4-bit model produces garbage, suspect the scale layout before the model.

GGUF K-Quants: The Byte Layouts, Verified

llama.cpp documents the arithmetic of its k-quants. Its source defines the super-block structure (QK_K = 256) and carries static_asserts for the byte counts; these are the per-weight bit costs:⁷

Read Q4_K's row as the generic formula again: 2 bytes d + 2 bytes dmin + 12 bytes packed 6-bit scales + $(256 \times 4)/8 = 128$ bytes of nibbles = 144 bytes per 256 weights — $144 \times 8 / 256 = 4.5$ bits per weight, plus sixteen levels with a two-level scale hierarchy (super-block FP16 scale over per-block 6-bit scales) so fine scaling doesn't cost a full FP16 per 32 weights.

The _S/_M/_L suffixes describe a tensor policy: they pick which tensors get stored above base width (Q4_K_M keeps attention.v and MLP down-proj at higher precision than Q4_K_S). Published effective bpw varies by model: Q4_K_M measures 4.84 bpw on Llama-2-7B, 4.83 on 13B and 4.80 on 70B in llama.cpp's table.⁷ The repository's perplexity measurements are specific to the tested LLaMA checkpoints and do not establish a universal quality penalty for another model or task.

Putting the Formats Side by Side

Effective bits and what you pay for them, using $b_{\text{eff}}$ from the group-overhead formula (integer formats at typical group 128; NVFP4 at 4.50; GGUF at published bpw):

Three honest observations fall out of the table:

1. The example 4-bit widths span about 17.7–20.2 GB for 33.4B weights, before unquantized tensors, KV cache and runtime allocations. The site calculator's NVFP4 entry uses 4.5 bpw, which already includes the FP8 block scales; it does not add another 10%.
2. FP8 halves the raw payload relative to FP16/BF16, but scales and unchanged tensors add bytes. Quality depends on the model, checkpoint and task.
3. The relative accuracy of NVFP4, MXFP4, AWQ and GGUF quantizations must be measured on the same model and evaluation set. NVIDIA's published FP8-versus-NVFP4 results describe its specific conversion study, not a universal 1% bound.

Which Format Should You Run?

Self-hosted, single-user or small-batch (llama.cpp/LM Studio/Ollama): start Q4_K_M. Size any step-up with $P \times b_{\text{eff}}/8$ against your VRAM after reserving KV-cache (the break-even calculator's KV math applies unchanged). If code/math quality matters, Q6_K; if you are establishing a reference, Q8_0.

vLLM/SGLang/TensorRT-LLM serving on Hopper: FP8 (E4M3 weights, per-tensor or per-block scales per your runtime's support). 4-bit integer weights (AWQ/GPTQ) when VRAM pressure demands it — expect a measurable but small quality step.

Blackwell serving at scale: consider NVFP4 when the checkpoint and runtime support it. It is native to Blackwell's FP4 tensor cores; NVIDIA reported close accuracy to FP8 in a particular DeepSeek conversion study, which should be reproduced on your task. MXFP4 is an OCP standard with a different 4.25-bpw block layout.

Whatever you pick, measure. Publish your own before/after on your workload — the perplexity deltas above are Llama-family measurements under llama.cpp calibration; arrival distributions, long-context regimes, and tool-calling patterns shift them. The formulas in this article give you the frontier; only your benchmark tells you where on it to sit.

Further Reading

• Nemotron 3.5 Bolt: heterogeneous LatentMoE and IQ2 — decode the 66-byte IQ2_XXS layout and separate mixed-precision weights from hybrid runtime state.
• LLM VRAM Requirements: A Mathematical Deep Dive — the full weight + KV-cache memory model this article's examples plug into.
• LLM Inference Math: From Theory to Hardware — why decode is memory-bound, which is why $b_{\text{eff}}$ is also your throughput knob.
• How LLMs Actually Work — tokenizers, attention and sampling background for the format decisions above.