Lesson 02 — what actually happens when a model emits one token. After this lesson you should be able to narrate the forward pass without hand-waving. ## One token. Seven steps. Every token a large language model produces comes from the same seven-stage pipeline. Skip any one of them and you lose the ability to reason about cost, latency, quality, or failure. The point of this lesson is that those seven stages stop being words and start being a thing you can picture. Use the block below to step through one forward pass on the sentence `"The cat sat on the"`, ending at the predicted next token. 01 · Tokenise [5] int 02 · Embed [5, 4096] 03 · Transformer block (× N) [5, 4096] 04 · Final norm + head [5, 128256] 05 · Softmax [128256] 06 · Sample [1] int 07 · Append + repeat [6] int The #791 cat #8415 sat #7731 on #389 the #279 5 tokens. The model will see only the numbers above each token. The [4096] cat [4096] sat [4096] on [4096] the [4096] 5 vectors of length 4096. Each vector is a row of the embedding table. 1. RMSNorm stabilise the activations 2. Self-attention Q, K, V projections; RoPE rotates Q/K; mask future positions; softmax(QKᵀ/√d) · V 3. Residual add input back in 4. RMSNorm 5. MLP two linear layers with SwiGLU activation, expanding ~3.5× then back 6. Residual One block. Then 31 more like it. KV cache stores K, V at each position so future tokens don’t recompute them. mat 7.63 floor 6.73 bed 6.20 chair 5.84 couch 5.50 table 5.28 grass 4.99 lap 4.59 Top candidates by raw logit. Notice they’re unbounded real numbers, not yet probabilities. mat 42% floor 17% bed 10% chair 7% couch 5% table 4% grass 3% lap 2% After softmax. All probabilities sum to 1 across the full 128k vocab; the top 8 shown. mat 42% floor 17% bed 10% chair 7% couch 5% table 4% grass 3% lap 2% Greedy sampling: the highest-probability token wins. With temperature > 1 the distribution flattens and lower-probability tokens become more likely. The #791 cat #8415 sat #7731 on #389 the #279 mat new The new token joins the input. KV for positions 1–5 is cached; the next forward pass only computes attention for position 6. ← Previous Next → Restart ## The two operations that matter Inside each transformer block, two things happen that have very different shapes. Knowing which is which is the difference between an architect and someone who can recite block diagrams. - **Self-attention** mixes information *across positions*. Each token decides how much to pay attention to every previous token and pulls a weighted sum of their value vectors back to itself. This is where context gets read. - **MLP** transforms each position *independently*. No mixing. It takes the post-attention vector for a single token and runs it through two linear layers with an activation in the middle. This is where most of the parameters live. That asymmetry is why long context is expensive (attention is quadratic in sequence length) but model capacity feels generous (most params are in the position-independent MLP). ## The numbers, for one real model Llama-3-8B, the everyday workhorse: - **Vocab**: 128,256 tokens - **Hidden dim**: 4,096 - **Layers (transformer blocks)**: 32 - **Attention heads**: 32 query heads, 8 KV heads (so it’s grouped-query attention with a 4:1 ratio) - **Head dim**: 128 - **MLP hidden**: 14,336 (roughly 3.5× the residual stream, with SwiGLU) Most of the 8B parameters live in those 32 MLPs. The attention layers are smaller in parameter count but bigger in compute during inference because of the quadratic seq_len dependency. ## Why you’ve now done Layer 1’s eval gate The Layer 1 evaluation is: *explain text → next token, without hand-waving*. Reading the steps in the block above gives you that pipeline: 1. Tokenise the string into integer IDs. 2. Look up each ID in the embedding table to get vectors of size `hidden`. 3. Send those vectors through `N` transformer blocks. Each block normalises, runs causal self-attention with RoPE-rotated Q/K, adds the residual, normalises, runs the MLP, and adds the residual. 4. After the last block, one more norm, then project the final position’s hidden vector to a logit vector of length `vocab`. 5. Softmax the logits to get a probability over every possible next token. 6. Sample one — greedily, or with temperature, top-k, top-p. 7. Append the new token, jump back to step 3 with the position extended by one (and the previous K/V cached). The remaining lessons in this track unpack the bits the walkthrough glossed over: *sampling* (lesson 03) shows what happens between logits and the picked token, and *KV cache* (lesson 04) shows why the autoregressive loop in step 7 is the most operationally important fact about LLM serving. --- *Numbers in the walkthrough use Llama-3-8B as the worked example. The probability distribution shown in steps 4–6 is illustrative; in practice you’d never see only 8 candidates — the softmax runs over the full 128k vocab and almost all of the mass concentrates on a handful of plausible tokens.*