Skip to content
Friendly disclaimer: flozi00 TechHub is a solo side-project next to a full-time job — personal learning notes, no official statements. Verify critical steps yourself.
flozi00 Logoflozi00 TechHub
ENDE

The KV Cache Explained Visually

See what a language model saves in its KV cache, how prompt processing differs from generation, and why longer context uses more memory.

2 min readflozi00
aibeginnervisual-learningkv-cacheinferenceguide

A KV cache saves previously computed attention keys and values so the model can use them again. This avoids repeating work when a language model generates another token. The cache stores numerical intermediate results for a sequence.1

If Q, K and V are still unfamiliar, start with the attention lesson. The useful idea is simple: a new query compares with keys and uses the corresponding values to build a mixture.

Watch · explore · understand

KV cache: keep the work you already did

Follow one prompt through prefill and one decode step. Watch what is saved, what is new, and what attention still has to read.

AI-assisted learning visualization · simplified worked example

Step 1 / 6Read the prompt

Our example prompt has three tokens. The model has not processed them yet, so this request’s KV cache is empty.

Simplified causal decoder · one layer shown

Input to this pass

Theskyis

3 prompt tokens

Learned projections
Q…
K…
V…

Not calculated yet

Saved K/V pairs

0 positions

1. The

——

Empty

2. sky

——

Empty

3. is

——

Empty

4. …

——

Empty

Word labels identify positions. Each K/V box holds numerical vectors, not the word or a saved answer.

Read the cacheCompare Q with K → attention weights → mix V

Attention runs when an input token is processed. Sampling alone does not extend the cache.

Generated output so farNo output token chosen yet

Simplified example: each word stands for one token here. Real tokenizers can split words. The continuation is scripted, not a model prediction.

Why is reuse valid?

For a fixed prefix in causal inference, earlier token states cannot depend on future tokens. Their computed keys and values can therefore be reused. Each layer keeps its own cache; the model weights do not change.

Why does a longer conversation use more memory?

Try the raw K/V payload for a hypothetical full-attention model. Doubling stored tokens doubles this payload.

Raw K/V payload

512 MiB

128 KiB per stored token

0–8 GiB

1 sequence · 32 layers · 32 query heads · 128 numbers per head · BF16 cache (2 bytes per number). These are example dimensions, not a named model.

Show the calculation and its limits

Bytes = 2 × sequences × layers × tokens × KV heads × head dimension × bytes per number

2 × 1 × 32 × 4,096 × 8 × 128 × 2 = 536,870,912 bytes

The factor 2 accounts for keys and values. GQA and MQA store fewer K/V heads; query heads are separate.

This estimate excludes model weights, temporary activations, reserved capacity, paging and allocator overhead. Sliding-window, compressed and hybrid caches need different estimates. Cache precision is independent of weight precision.

Sources: Hugging Face · How caching works · NVIDIA · KV cache

Set your own pace. Pause at any time or select a step directly.

Follow one generation cycle

  1. Read the prompt — prefill. The model processes the input tokens and builds their keys and values at each attention layer. The final prompt position provides the prediction used to choose the first output token.
  2. Process the new token — decode. That chosen token becomes the next input. Each layer computes its new query, key and value.
  3. Reuse and append. The new query reads the earlier cached keys together with the current key. Attention mixes their values, and the current K/V are retained for the following step.
  4. Repeat. The model predicts another token, which can be fed back in. A conventional full-attention cache grows as more positions are processed.1

Why is reuse possible?

In a causal decoder, an earlier position cannot read future tokens. With the prefix and model unchanged, adding a later token therefore does not require recomputing those earlier keys and values. Each layer has its own cached tensors; keys from one layer are not a substitute for keys from another.1

In the diagram, follow the older entries as a new entry arrives. The important distinction is between reusing stored vectors and calculating the new token's attention. The second job still happens.

The memory tradeoff

More stored tokens mean more K/V entries. More simultaneous sequences, layers, KV heads or bytes per entry also increase the raw storage required. Actual runtimes may reserve extra capacity or organize the cache into blocks.2

For a small counting example, four tokens stored as two K numbers and two V numbers each require 16 numbers. A fifth token adds four more. Real models repeat this across heads and layers, so the same growth can become a substantial memory cost.

Caching saves recomputation, but full attention still reads a growing set of keys and values during decode. It does not make the work per generated token independent of context length.1

Avoid this misunderstanding

The KV cache is not a collection of finished answers or newly learned model weights. Editing an earlier part of the prompt generally requires rebuilding the affected states. This lesson illustrates ordinary causal decoder inference; sliding windows, compressed caches and other architectures can use different layouts.1

Next: learn how NVFP4 and INT4 store numbers. For more depth, open the KV-cache glossary or return to the learning path.

Sources

Footnotes

  1. Hugging Face, How caching works. ↩ ↩2 ↩3 ↩4 ↩5

  2. NVIDIA TensorRT, KV cache tensor layouts and memory ownership. ↩