Skip to content
Friendly disclaimer: flozi00 TechHub is a solo side-project next to a full-time job — personal learning notes, no official statements. Verify critical steps yourself.
flozi00 Logoflozi00 TechHub
ENDE

Attention Explained Visually

Follow queries, keys, values and softmax through a small attention example, with step-by-step diagrams and plain-language explanations.

2 min readflozi00
aibeginnervisual-learningattentiontransformerguide

Attention lets a token's representation combine information from other positions in the text. Imagine reading “The cup fell because it was unstable.” Interpreting “it” requires context. Attention provides a numerical way to mix context; it does not contain a handwritten rule that identifies pronouns.1

A token is a piece of input handled by the model; it may be a word or part of one. A vector is simply a list of numbers. This lesson follows one attention head, one of several parallel calculations inside a Transformer layer.

Watch · explore · understand

Attention: build a useful view of the context

Follow one token as it compares, weighs and combines information from earlier tokens.

AI-assisted learning visualization · simplified worked example

Step 1 / 5Make Q, K and V

A token is represented by a list of numbers: a vector. Three learned transformations produce a query Q, a key K and a value V. The query is what this position uses to look for matching information.

Which token is looking?

Select a token. The calculation updates at every step.

Selected query: “sat”Q = [1.5; 1]

Representation: X = [1; 1]

Q · what to match
[1.5; 1]
K · what to match against
[1; 1]
V · information to combine
[1; 2]
  1. 1 · TheVisible context
    Key K[1; 0]
    Value V[1; 0]
  2. 2 · catVisible context
    Key K[0; 1]
    Value V[0; 2]
  3. 3 · satThis position
    Key K[1; 1]
    Value V[1; 2]
  4. 4 · .Later position · masked in step 3
    Key K[0.5; -0.5]
    Value V[0.5; -1]

The value mix appears in step 5.

Teaching example: invented two-dimensional vectors. Words stand in for tokens; real token boundaries can differ. The numbers do not measure the meaning of this sentence. Displayed numbers are rounded; calculations use full precision.

Show the formula and the example matrices

softmax(QKᵀ / √dₖ + M) V

Q = XWQ, K = XWK, V = XWV. Here dₖ = 2. The mask M is 0 for allowed positions and −∞ for future positions. Softmax is applied over the keys for this one query.

These fixed toy matrices stand in for parameters learned during training. No training happens in this demo.

WQ[1; 0]
[0.5; 1]
WK[1; 0]
[0; 1]
WV[1; 0]
[0; 2]

Scope: one head of full causal self-attention. Positional encoding, other heads, output projection, residual connections, normalization and feed-forward layers are omitted.

Source: Attention Is All You Need, section 3.2
Set your own pace. Pause at any time or select a step directly.

The whole calculation in plain language

  1. Create three views. Learned transformations turn each token's current vector into a query (Q), key (K) and value (V). Think of Q as what this position looks for, K as what another position can match, and V as the information it can contribute. These are analogies for numerical vectors.
  2. Compare Q with K. Multiplying corresponding entries and adding the results produces a score. A scaling factor accounts for the number of entries in Q and K.
  3. Hide unavailable positions. In causal self-attention, a position can use itself and earlier positions. Later positions are excluded before the next step.
  4. Turn scores into weights. Softmax makes the allowed weights nonnegative and makes them sum to one. A higher score receives a larger share.
  5. Mix V. Multiply each value vector by its weight and add the results. The result is a new vector containing a weighted mixture.12

Read the diagram actively

Follow one position through the sequence. At the scoring step, ask which comparisons are allowed. At the weighting step, check that the displayed shares add up to approximately 100%; rounded labels can differ slightly. At the mixing step, notice that several positions can contribute at once.

For a separate tiny example, suppose two scalar values are 2 and 10. Weights of 75% and 25% produce 4: three quarters of 2 plus one quarter of 10. The lesson performs the same kind of mixture with lists of numbers.

What this example does and does not show

The vectors and scores are teaching examples, not measurements from a trained model. Position handling and the rest of the Transformer are simplified. The original Transformer combines multiple heads and adds further processing around attention.1

An attention weight is not the probability of the next word. It controls a contribution to an intermediate representation. Predicting the next token requires the rest of the model and a separate output over its vocabulary.1

Continue

Next: see why the KV cache keeps keys and values. For the surrounding model, read How LLMs Actually Work, or return to the visual learning path.

Sources

Footnotes

  1. Vaswani et al., Attention Is All You Need, sections 3.2–3.4. ↩ ↩2 ↩3 ↩4

  2. PyTorch, Scaled dot-product attention: reference computation and masking. ↩