How LLMs Actually Work — An Animated Walkthrough

Start with the building blocks: Explore attention step by step, then follow KV-cache reuse. The visual learning path connects all three beginner lessons. The animation follows the prompt "The cat sat on the mat because it was" to an illustrative next-token choice. Its word chunks, attention links and probabilities are invented for teaching; they are not output from a measured model.

The one-sentence version

An autoregressive language model maps a sequence of tokens to next-token scores. Softmax can turn those scores into a probability distribution; repeated selection generates text. This explains the generation loop, though actual assistants also involve tokenization, multimodal inputs, post-training, tools and runtime orchestration.

If you internalize that sentence, the rest of this page is just mechanical detail about how the function is built.

Stage by stage

From letters to IDs

The model consumes token IDs produced by a tokenizer. Many LLM tokenizers use BPE variants, but vocabulary and token boundaries vary by model; a word may be one or several tokens. The animation colors whole words only to make the stages readable. Token counts for the same text can differ substantially across tokenizers and languages. See the Hugging Face tokenizer documentation.

One consequence that surprises people: the model has no built-in concept of a "word". Spelling tricks, homoglyphs, or unusual spacing can change the token stream — and therefore the computation — even when the meaning for a human is identical.

From IDs to geometry

Each token ID indexes a learned embedding vector. Contextual representations across later layers can capture semantic relationships; the particular nearest neighbors of raw input embeddings are model-dependent and are not measured in this animation.

Position enters the computation too. In models using rotary position embeddings (RoPE), position rotates query and key vectors within attention; it is not simply a vector added to each input embedding.

Attention: the actual idea

Attention is the load-bearing concept. For every token, the model computes three projections of its vector — a query, a key, and a value — and then:

1. the query of token i is dotted against the keys it may see (in full causal self-attention, itself and earlier tokens),
2. the scores are scaled and softmaxed into weights that sum to 1,
3. the output is the weighted mixture of the values.

This is a weighted mixture of value vectors, as in the original Transformer paper. The thick line from “it” to “cat” is a possible teaching example; those weights were not measured, and a single attention weight is not a reliable explanation of a model's reasoning.

Full self-attention performs O(N²) query-key comparisons over an N-token prefill. Sliding-window attention changes which pairs are compared. IO-aware implementations such as FlashAttention reduce memory traffic without removing all of that arithmetic. During generation, KV-cache storage grows with the number of cached positions; grouped-query attention and KV quantization can reduce bytes per position.

Why depth beats width

Transformer blocks combine attention, feed-forward layers, normalization and residual connections; block count and order differ across models. Some mechanistic studies find features such as induction heads, but these findings do not assign a fixed role to every layer of every LLM.

This is also where the parameters live: the block's matrices scale with the square of the hidden size, so doubling a model's width roughly quadruples its parameters — which is why a 70B model needs ~140 GB of weight storage at BF16 where a 4B model needs ~8 GB (2 bytes per parameter, linear in parameter count). The LLM inference math guide covers the resulting memory and compute budgets, and the Qwen3-Next notes show how sparse MoE activation bends the rule.

Sampling: where "creativity" hides

The final hidden vector of the last position is projected to vocabulary scores, and softmax can convert them to probabilities. Sampling introduces variation; a fixed seed can reproduce outputs only when the rest of the numerical execution is deterministic:

• Temperature rescales logits before softmax. Lower temperatures sharpen the distribution; higher temperatures flatten it. Neither setting guarantees safety or correctness.
• Top-p / top-k truncate the distribution before sampling, cutting the long tail of nonsense tokens.

The animation's stage 6 shows exactly this: the same context, a distribution over candidate next tokens, and one sampled winner.

Training: predict, correct, repeat

Pre-training is embarrassingly simple at heart: show the model a chunk of text, ask it to predict each next token, measure how wrong it was (cross-entropy), and nudge all the matrices a tiny step in the direction that reduces the error (backpropagation + an optimizer). Repeat over trillions of tokens. There is no teacher that explains why — only the raw pressure of the next token.

Post-training may include supervised instruction fine-tuning and preference optimization such as RLHF or DPO. These methods shape behavior, but neither helpfulness nor factual accuracy follows automatically from them.

What the animation leaves out

Honesty pass — the parts the pretty version skips:

• KV cache & decode mechanics. Real generation caches keys and values so each new token doesn't reprocess the whole prompt; that cache dominates serving memory at long context. The inference calculator lets you compute it.
• MoE routing. Many current models (including Qwen3-Next) route each token through a subset of experts; the six stages stay identical, only the feed-forward step becomes sparse.
• Multimodality. Images, audio and video may be encoded into model-specific token or embedding sequences; the encoders and fusion paths differ by architecture.
• Reasoning models. "Thinking" tokens are still the same next-token loop — just run longer before answering. The machinery in this article doesn't change.

A released example is Mellum2.1: each token uses eight of 64 experts, while 21 local and seven global attention layers handle different distance ranges. Only the global layers apply YaRN position scaling. The architecture guide follows its actual tensor shapes and explains why active parameters do not equal resident weight memory.

A pretrained language-model backbone can also feed a decision head. Cloudflare Clef Omni encodes text, audio and visual inputs, then scores declared schema options after one prefill. Its wrapper replaces the vocabulary projection with an identity module and retains the output-embedding weights for option lookups, so no next-token sampling or decode loop follows.

Liquid d1-3B takes a different route: it keeps the tied vocabulary head, packs one shared state as a trunk with independent question branches, and merges branch attention exactly from partial log-sum-exp statistics. The guide derives which work is actually shared and why its returned confidence is not calibrated by default.

If you want the performance side — how all this maps to FLOPS, bandwidth, and euros — continue with the inference economics page, which builds on the same pipeline from the hardware's point of view.

Related

• Nemotron 3.5 Super VL: the visual token pipeline — follow dense image features, paired video patches and change-based token pruning before the autoregressive language backbone.

• Beam 1.0: local RoPE, global NoPE and scalable softmax — follow position-dependent query scaling and derive why sharpening attention does not guarantee long-context recall.

• Muse Spark Open: residual gates and normalized experts — follow three different gate mechanisms and see why normalization order is part of model correctness.

• Nemotron 3.5 Bolt: heterogeneous LatentMoE and IQ2 — separate full-width routing and shared experts from a reduced-width routed branch, then inspect its mixed-precision storage.

• Step-5-Preview: sparse reads versus KV storage — follow eight-token index summaries, group-specific block selection and attention over original K/V.

• Ling-3.1-flash: grouped routing and MLA cache layouts — follow KDA/MLA tensor shapes and the algebra that moves key/value projections out of the request cache.

• LFM2.5-24B-A2B routing and cache architecture — follow top-four sigmoid routing and see why only ten layers accumulate KV while all expert weights remain resident.

• LLM VRAM requirements: a mathematical deep dive — size the weights + KV cache budget for any model.

• Choosing the right GPU for Qwen3 inference — match model footprints to hardware.

• The LLM inference math guide — the roofline formulas behind every number here.